Memory leak on executables embedded with 3.13

[neonene]

Bug report

Bug description:

Since 67807cf, _testembed.exe (x64 Release) increases the memory usage in the cycle of Py_Initialize and Py_Finalize.

I monitored the Task Manager (taskmgr.exe) on Windows10/11, with a change against Programs/_testembed.c:

- #define INIT_LOOPS 4
+ #define INIT_LOOPS 200

Commands:

• _testembed.exe test_repeated_simple_init or
• python -m test test_embed -m test_simple_initialization_api (no log)

With RADIX TREE (as-is)

Loop	67807cf	main	3.11.7
100	67MB	210MB	4MB
200	135MB	406MB	4MB

No RADIX TREE (as-is)

Loop	67807cf	main	3.11.7
100	10MB	168MB	4MB
200	16MB	336MB	4MB

Recent obmalloc.c looks ready for finalization. Just adding my rough (invalid?) experiment made the leaks even. So, I hope that they share the same issue. Otherwise, ea2c001 or 15d4c9f has another leak. #98359 (comment)

patch

void
_PyInterpreterState_FinalizeAllocatedBlocks(PyInterpreterState *interp)
{
    if (has_own_state(interp)) {
        Py_ssize_t leaked = _PyInterpreterState_GetAllocatedBlocks(interp);
        assert(has_own_state(interp) || leaked == 0);
        interp->runtime->obmalloc.interpreter_leaks += leaked;

+       OMState *state = &interp->obmalloc;
+       for (uint i = 0; i < maxarenas; ++i) {
+           if (allarenas[i].address == 0) {
+               continue;
+           }
+           _PyObject_Arena.free(_PyObject_Arena.ctx,
+                                (void *)allarenas[i].address, ARENA_SIZE);
+           allarenas[i].address = 0;
+           --narenas_currently_allocated;
+       }
+       PyMem_RawFree(allarenas);
    }
}

With RADIX TREE (patched)

Loop	67807cf	main
100	42MB	40MB
200	80MB	80MB

No RADIX TREE (patched)

Loop	67807cf	main
100	3.5M	3.3M
200	3.7M	3.4M

cc @ericsnowcurrently @nascheme

CPython versions tested on:

3.12, 3.13, CPython main branch: 3aea6c4

Operating systems tested on:

Windows

Linked PRs

• gh-113055: Fix memory leak of obmalloc state #113218
• gh-113055: Use pointer for interp->obmalloc state. #113412
• gh-118608: datetime: Fix use-after-free on embedded CPython #118531
• [3.12]: gh-113055: Use pointer for interp->obmalloc state #118618

[nascheme]

That doesn't look so nice, I can investigate. How are you measuring memory usage and what's the OS? In the previous versions of Python the obmalloc radix tree data structures were in the BSS and so most of RAM used by them was actually virtual and not real. I'm not sure if the move to interpreter state structures lost that behaviour but it could be cause for an increase in (real) memory use. Maybe there is also a memory leak there, e.g. malloc without free.

[neonene]

How are you measuring memory usage and what's the OS?

Only the Task Manager on Windows 10 and 11 for me.
The microsoft.com offers VMMap as a small alternative.

[nascheme]

@ericsnowcurrently

I spent some time looking into this today. I suspect the fix is not going to be simple, unfortunately. What was previously global variables in the obmalloc.c module has become part of the PyInterpreterState structure. When an interpreter is freed, memory allocated by obmalloc doesn't get freed. I think that's the leak you are seeing. Previous to 67807cfc it didn't get freed either but it was global across interpreters, got allocated only once and didn't grow.

A possible way to fix this: when free_interpreter() gets called, walk through obmalloc arenas and free them. The pymalloc_print_stats() function is an example of how to walk over the arenas. The obmalloc radix tree nodes would also have to be walked and deallocated.

If you free the arenas and there is allocated blocks inside of them, you can't call pymalloc_free() on those memory blocks. It is maybe not a problem since the interpreter is being destroyed. If an object is passed from one interpreter to another, that would definitely break. If you want to share objects across interpreters, they would need to be allocated from a memory allocator that's global across interpreters.

[nascheme]

Building gh-113218 with the address sanitizer and then running ./Programs _testembed test_repeated_simple_init shows it doesn't leak anymore. I'm not sure my fix is totally correct but I think it's something close to what we need to fix this.

[neonene]

_testembed test_repeated_simple_init shows it doesn't leak anymore

I can also personally confirm that on Windows, including the 32-bit version of Python.

[nascheme]

I think I've come up with a better fix. See gh-113412.

[neonene]

ea2c001 has a memory leak to be fixed, which is found at gh-113412.
I've seen a few reports about likely suspects: #110411 (comment), #113190 (comment)

[nascheme]

Your test_repeated_simple_init case leaks on Linux as well. With gh-113412 the leak is slower but it leaks without bound, it seems. Using valgrind, I suspect it (at least partially) due to immortal interned strings. I generated the trace with:

PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log --num-callers=20 ./Programs/_testembed test_repeated_simple_init

Trace log:

https://gist.github.com/nascheme/fac2dd566f09fe3d75c98134f18c6170

E.g.

==1275635== 502,840 bytes in 9,800 blocks are definitely lost in loss record 421 of 422
==1275635==    at 0x48407B4: malloc (vg_replace_malloc.c:381)
==1275635==    by 0x2A0F00: _PyMem_RawMalloc (obmalloc.c:56)
==1275635==    by 0x2A2568: PyObject_Malloc (obmalloc.c:917)
==1275635==    by 0x2EF536: PyUnicode_New (unicodeobject.c:1238)
==1275635==    by 0x2FBAC3: unicode_decode_utf8 (unicodeobject.c:4701)
==1275635==    by 0x2FC017: PyUnicode_DecodeUTF8Stateful (unicodeobject.c:4834)
==1275635==    by 0x2F0E38: PyUnicode_FromString (unicodeobject.c:1893)
==1275635==    by 0x3135D9: PyUnicode_InternFromString (unicodeobject.c:14943)
==1275635==    by 0x289EBA: PyObject_SetAttrString (object.c:1062)
==1275635==    by 0x28504F: _add_methods_to_object (moduleobject.c:174)
==1275635==    by 0x2857D1: PyModule_FromDefAndSpec2 (moduleobject.c:384)
==1275635==    by 0x3FB5C6: create_builtin (import.c:1396)
==1275635==    by 0x4002F6: _imp_create_builtin (import.c:3400)
==1275635==    by 0x284483: cfunction_vectorcall_O (methodobject.c:512)
==1275635==    by 0x21B736: _PyVectorcall_Call (call.c:273)
==1275635==    by 0x21B94C: _PyObject_Call (call.c:348)
==1275635==    by 0x21BA23: PyObject_Call (call.c:373)
==1275635==    by 0x3856F4: _PyEval_EvalFrameDefault (generated_cases.c.h:1250)
==1275635==    by 0x37E2B1: _PyEval_EvalFrame (pycore_ceval.h:115)
==1275635==    by 0x3AA774: _PyEval_Vector (ceval.c:1771)

And

==1275635== 339,080 bytes in 6,920 blocks are definitely lost in loss record 420 of 422
==1275635==    at 0x48407B4: malloc (vg_replace_malloc.c:381)
==1275635==    by 0x2A0F00: _PyMem_RawMalloc (obmalloc.c:56)
==1275635==    by 0x2A2568: PyObject_Malloc (obmalloc.c:917)
==1275635==    by 0x2EF536: PyUnicode_New (unicodeobject.c:1238)
==1275635==    by 0x2F11FE: _PyUnicode_FromUCS1 (unicodeobject.c:2025)
==1275635==    by 0x2F16F9: PyUnicode_FromKindAndData (unicodeobject.c:2096)
==1275635==    by 0x414B4E: r_object (marshal.c:1157)
==1275635==    by 0x414DC0: r_object (marshal.c:1222)
==1275635==    by 0x415AC6: r_object (marshal.c:1399)
==1275635==    by 0x414DC0: r_object (marshal.c:1222)
==1275635==    by 0x415A8A: r_object (marshal.c:1393)
==1275635==    by 0x416028: read_object (marshal.c:1524)
==1275635==    by 0x416339: PyMarshal_ReadObjectFromString (marshal.c:1641)
==1275635==    by 0x3FCDF0: unmarshal_frozen_code (import.c:2062)
==1275635==    by 0x4008B1: _imp_get_frozen_object_impl (import.c:3575)
==1275635==    by 0x3F8E6B: _imp_get_frozen_object (import.c.h:282)
==1275635==    by 0x284098: cfunction_vectorcall_FASTCALL (methodobject.c:425)
==1275635==    by 0x21B736: _PyVectorcall_Call (call.c:273)
==1275635==    by 0x21B94C: _PyObject_Call (call.c:348)
==1275635==    by 0x21BA23: PyObject_Call (call.c:373)

[nascheme]

The following change seems to greatly reduce the leaks. The _Py_hashtable_set(INTERNED_STRINGS, s, s) call looks like a problem to me.

--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -14870,6 +14870,7 @@ _PyUnicode_InternInPlace(PyInterpreterState *interp, PyObject **p)
         return;
     }
 
+#if 0
     /* Look in the global cache first. */
     PyObject *r = (PyObject *)_Py_hashtable_get(INTERNED_STRINGS, s);
     if (r != NULL && r != s) {
@@ -14885,6 +14886,7 @@ _PyUnicode_InternInPlace(PyInterpreterState *interp, PyObject **p)
         }
         return;
     }
+#endif
 
     /* Look in the per-interpreter cache. */
     PyObject *interned = get_interned_dict(interp);