Increase fraction of code executed by tier 2.

[markshannon]

According to stats and profiling only about 40% of bytecode instructions are executed by tier 2 and the remaining 60% by tier 1.

We the expected improvements to the JIT and tier 2 optimizer we expect tier 2 (with JIT) to have a significantly faster than tier 1.
It therefore make sense to get the fraction of instructions executed by tier 2 up from 40% to nearer 90%.

To do that we need to:

• Enter tier 2 at some function entry points as well as back edges. #118094
• Stay in tier 2 when exiting executors in most cases Trace stitching #112354
• Increase the number of micro-ops that we can handle in tier 2 #118095

Linked PRs

• GH-118093: Add tier two support to several instructions #121884
• GH-118093: Remove invalidated executors from side exits #121885
• GH-118093: Better handling of short and mid-loop traces #122252
• GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE #122253
• GH-118093: Add tier two support for LOAD_ATTR_PROPERTY #122283
• GH-118093: Handle some polymorphism before requiring progress in tier two #122843
• GH-118093: Turn some DEOPT_IFs into EXIT_IFs #122998
• GH-118093: Specialize CALL_KW #123006
• GH-118093: Specialize CALL_FUNCTION_EX #123034
• GH-118093: Make CALL_ALLOC_AND_ENTER_INIT suitable for tier 2. #123140
• GH-118093: Specialize calls to non-vectorcall classes as CALL_NON_PY_GENERAL #123212
• GH-118093: Fix off-by-one errors in tier-up thresholds #124447
• GH-118093: Don't lose confidence when tracing through 100% biased branches #124813

[brandtbucher]

By my count we're currently hovering around 54% of code executed in tier two (our benchmarks run about 266 billion tier one instructions on normal builds and 122 billion instructions on JIT builds). I've identified a few strategies for improving this (based on stats and tracing through how we execute a bunch of the benchmarks) and will start landing PRs soon. No magic bullets here, just chipping away at things:

• Add specializations for CALL_KW and CALL_FUNCTION_EX. We probably don't need to do anything too crazy here early on in terms of optimizing calls... we can start by just adding a handful of specializations that allow us to trace through them instead of ending the trace.
• Add tier two support to several other instructions that are prematurely ending traces. Some of these are easy (CALL_LIST_APPEND, IMPORT_NAME, LOAD_NAME, BUILD_SET, SEND_GEN, and IMPORT_FROM), and some are harder (LOAD_ATTR_PROPERTY, BINARY_SUBSCR_GETITEM, CALL_ALLOC_AND_ENTER_INIT, RAISE_VARARGS, and BINARY_OP_INPLACE_ADD_UNICODE).
• Specialize SEND_ASYNC_GEN_ANEXT, using a similar shim frame as CALL_ALLOC_AND_ENTER_INIT.
• Handle underflow, either dynamically (with DYNAMIC_EXIT) or statically (by using the current stack when projecting to infer callers). A more radical idea could be to start recording traces instead of projecting them. That simplifies a lot of things, but is a big rewrite of some pretty core stuff.
• Turn more DEOPT_IFs into EXIT_IFs for better handling of control flow and polymorphism. _FOR_ITER_TIER_TWO is an obvious candidate here, but there are others, too.
• Allow shorter traces (it's tricky to do this while still requiring progress, but doable).
• Better handling of polymorphism (our current progress requirement inhibits this, but that can be relaxed without too much trouble).
• Remove invalid traces from side exits (currently we just remove them from the bytecode, and side exits not only keep the invalid trace alive and continually deopting, but also prevent new traces from taking their place).
• Be better about closing loops in one trace, by allowing a single jump to occur anywhere in a trace, rather than always at the start.

My motivation for this is to make JIT improvements more pronounced. We currently spend less than 10% of our time in the JIT (vs ~25% of our time in tier one), which means that we need to improve the performance of JIT code by over 10% just to see a 1% improvement on the benchmarks. My (probably ambitious) goal is to get the fraction of code executed in tier two up to around 80% (meaning, in the neighborhood of 25%-30% of the total time spent running the benchmarks) in the next couple of weeks. Then the improvements can be easier to measure and iterate on.

[brandtbucher]

It's also worth noting that our stats are currently broken on benchmarks that use C extensions or spawn subprocesses. So the actual numbers may vary a bit right now, but probably aren't heavily biased one way or another.