PYTHONMALLOCSTATS=1 fails with fatal error at Python exit

[vstinner]

Py_FinalizeEx() calls _PyObject_DebugMallocStats() which calls indirectly _PyInterpreterState_GET(). Problem: at this point, there is no "interpreter" anymore, and so _PyInterpreterState_GET() fails with a fatal error.

It's a regression introduced by commit df3173d. Python 3.12 is also affected.

Reproducer:

$ PYTHONMALLOCSTATS=1 ./python -c pass
Small block threshold = 512, in 32 size classes.
(...)
Total                              =              262,144

Small block threshold = 512, in 32 size classes.
(...)
Total                              =              655,360

Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, after Python initialization and before Python finalization, but the GIL is released (the current Python thread state is NULL)
Python runtime state: finalizing (tstate=0x0000000000ae7550)

Abandon (core dumped)

cc @ericsnowcurrently

Linked PRs

• gh-111499: Fix PYTHONMALLOCSTATS at Python exit #120021
• [3.13] gh-111499: Fix PYTHONMALLOCSTATS at Python exit (GH-120021) #120022
• [3.12] gh-111499: Fix PYTHONMALLOCSTATS at Python exit (#120021) #120023

[vstinner]

Apparently, nobody cares nor uses this feature which has no test. I just close the issue. I don't have the bandwidth to work on a fix.

[ShahriyarR]

Not sure if this is the related information or not, added here for future reference.
I got the same error with Python 3.12 while doing some tests.
Compiled Python as described here:
https://devguide.python.org/

Test:

$ gdb python

(gdb) break main
(gdb) call PyLong_FromLong(22222)
Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, after Python initialization and before Python finalization, but the GIL is released (the current Python thread state is NULL)
Python runtime state: unknown

Basically, the fix is calling Py_Initialize() before:

(gdb) call Py_Initialize()

The difference is, in Python 3.9 I got sissegv instead of this Fatal error.

[devdanzin]

Apparently, nobody cares nor uses this feature which has no test. I just close the issue. I don't have the bandwidth to work on a fix.

I'm trying to add a test as part of #120006, but having segfaults (Linux) or non-zero process exit codes (Windows) is kinda getting in the way. Could this issue be reopened so other devs can attempt to solve it?

It seems the error has changed between versions:

Python 3.12.3 (Linux), non-debug build:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7ac632a in _PyInterpreterState_GET () at ./Include/internal/pycore_pystate.h:133
133         return tstate->interp;

Python 3.14.0a0 (heads/main:6e012ced6c) (Linux), non-debug build:

Program received signal SIGSEGV, Segmentation fault.
0x00005555556b0e3c in get_state () at Objects/obmalloc.c:1368
1368        return interp->obmalloc;

The fix might be as simple as (pardon my C):

diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
index 4fe195b631..04e24bd407 100644
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -3266,6 +3266,15 @@ py_mimalloc_print_stats(FILE *out)
 static void
 pymalloc_print_stats(FILE *out)
 {
+    PyThreadState *tstate = _PyThreadState_GET();
+    if (tstate == NULL || tstate->interp == NULL) {
+        return;
+    }
+
+    PyInterpreterState *interp = _PyInterpreterState_GET();
+    if (interp == NULL || interp->obmalloc == NULL) {
+        return;
+    }
     OMState *state = get_state();

     uint i;

[vstinner]

I wrote PR gh-120021 to fix the issue.