test__xxsubinterpreters is Occasionally Crashing

[ericsnowcurrently]

This appears to have started once gh-104210 was merged.

"Fatal Python error: drop_gil: PyMUTEX_LOCK(gil->switch_mutex) failed"

It may be specific to test.test__xxsubinterpreters.RunStringTests.test_create_thread.

See:

• https://buildbot.python.org/all/#builders/484/builds/3261
• GH-104308: socket.getnameinfo should release the GIL #104307 (comment)

Linked PRs

• gh-104341: Ensure tstate Holding GIL is Always Current #104436
• gh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization #104437
• gh-104341: Clean Up threading Module #104552
• gh-104341: Fix threading Module Shutdown #104560
• gh-104341: Minor Fixes in the _thread Module #104595
• gh-104341: Wait Completely at threading._shutdown() #104672
• [reverted] gh-104341: Add a Separate "Running" Lock for Each Thread #104754
• [3.12] gh-104341: Add a Separate "Running" Lock for Each Thread (gh-104754) #104817
• gh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizing the Current Thread #105109
• [3.12] gh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizing the Current Thread (gh-105109) #105209

[ericsnowcurrently]

CC @brandtbucher

[ericsnowcurrently]

I've been able to reproduce the crash locally with ./python -m test -j8 test__xxsubinterpreters --forever

[brandtbucher]

Core was generated by `./python -m test --forever test__xxsubinterpreters -m test_create_thread'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f44daa8a700 (LWP 23889))]
(gdb) bt
#0  raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000055d7555a0d94 in faulthandler_fatal_error (signum=11) at ./Modules/faulthandler.c:330
#2  <signal handler called>
#3  drop_gil (ceval=0x7f44dae104d8, tstate=tstate@entry=0x55d756081200) at Python/ceval_gil.c:301
#4  0x000055d75552fad5 in _PyEval_ReleaseLock (tstate=tstate@entry=0x55d756081200) at Python/ceval_gil.c:629
#5  0x000055d75556a9bd in _PyThreadState_DeleteCurrent (tstate=tstate@entry=0x55d756081200) at Python/pystate.c:1568
#6  0x000055d755615cb8 in thread_run (boot_raw=boot_raw@entry=0x7f44dab23790) at ./Modules/_threadmodule.c:1097
#7  0x000055d755587672 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:233
#8  0x00007f44dc5af609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00007f44dc37c293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[brandtbucher]

Also when running test_threads_join and test_threads_join_2 in test_threading with the same flags.

[ericsnowcurrently]

I think I've got this figured out. It looks like the interpreter is getting finalized before the thread is completely done in thread_run() (in _threadmodule.c). Apparently the thread isn't completely finalized at the point thread.join() returns. Thus there's a small race there where the runtime may think the thread is done running. It proceeds to clean up the thread state even though the thread is still cleaning itself up.

Based on the above, there are two points at which it should have waited:

• thread.join()
• wait_for_thread_shutdown() via Py_EndInterpreter() (in pylifecycle.c)

I'm going to figure out why things didn't wait properly. I suspect it looks something like this:

1. the sub-thread finishes
2. it "marks" itself as done
3. it releases the subinterpreter GIL
4. the subinterpreter's initial thread acquires the GIL
5. it finishes running
6. it releases its GIL
7. the sub-thread acquires the subinterpreter GIL
8. it finishes cleaning up its thread state
9. the main interpreter starts running again (with its own GIL)
10. it calls Py_EndInterpreter()
11. the thread is already off the "known threads" list so it doesn't wait there
12. it clears/frees the subinterpreter's remaining thread states and then the subinterpreter itself

The race lies between steps (8) and (12). If (8) doesn't finish before (12) happens then we'd see the crash we're seeing.

What I don't understand is:

• what do we need to do differently in step (2)?
• when is step (3) happening?

To be honest, I still haven't figured out what we are actually doing in (2) such that it affects thread.join() and wait_for_thread_shutdown(). However that may be irrelevant. I suspect we are missing a step before (9) where we acquire the subinterpreter's GIL while we're cleaning things up in Py_EndInterpreter().

FYI, this doesn't happen when the GIL is shared because (9) is blocked by the GIL being held still in the sub-thread.

[ericsnowcurrently]

This may also be related to (or a variant of) gh-96387.

[ericsnowcurrently]

Hmm, looks like I need to add _PyInterpreterState_GetFinalizing(), etc. and use it in tstate_must_exit() (in ceval_gil.c).