dict() resize failure

[kumbera1]

Bug report

When running under python 3.11.1, 3.11.2 and 3.12.a6, the dict() data structure fails to resize past 357913941 elements. This worked last (in what I have available) under python 3.10.8 and I can find no documentation that limits the size of a dict to a finite size. The system has >512G of memory and this example easily fits in that memory.

My reproducer is mem.py:

d=dict()
for a in range(500000000):
    try:
       d[a]=1
    except:
        print(a)
        exit(0)

output:
357913941

As an aside (2^31)/3/2 = 357913941. It looks to me like there is an int32 in the code that's trying to represent the size of the dict. (GROWTH_FACTOR=3) I'm not sure where the extra factor of 2 is coming from yet.

Simplest reproducer is:

d=dict()
for a in range(500000000):
       d[a]=1

output:
d[a]=1
~^^^
MemoryError

exit code=1

Your environment

I have tested on 3.10.8 (no failure) 3.11.1, 3.11.2, 3.12.a6 (fail)
This test must be run on a 64bit system with >40G of RAM free. (I have not tested on a system with less memory but I assume an Out of Memory error from the OS will stop the python program before the dict() failure>)

To reproduce I built the python with:

./configure --prefix=/mydir/python-3.12.a6 --enable-ipv6 --enable-shared --with-system-ffi --with-system-expat --with-ssl-default-suites=openssl --enable-optimizations
make -j
make install
set PATH=/mydir/python-3.12.a6/bin:$PATH and LD_LIBRARY_PATH=/mydir/python-3.12.a6/lib
python3 mem.py

In pyconfig.h
SIZEOF_VOID_P = 8
SIZEOF_SIZE_T = 8

OS:
Red Hat Enterprise Linux release 8.7 (Ootpa)

cat /proc/meminfo
MemTotal: 263686408 kB
MemFree: 150322584 kB
MemAvailable: 221228188 kB
...

Fails with both Intel and AMD servers.

Linked PRs

• gh-102701: Fix overflow in dictobject.c #102750
• [3.11] gh-102701: Fix overflow in dictobject.c (GH-102750) #102777

[kumbera1]

slightly smaller reproducer:

d = dict.fromkeys(range(357913941))
# dies with this line
d[357913942]=None

[methane]

Thank you for simple reproducer.
I think this is regression by f8a95df

[hauntsaninja]

Thanks for the fix and the great bug report!