Datetime NoneType after calling Py_Finalize and Py_Initialize

[DennyWeinberg]

BPO	27400
Nosy	@ncoghlan, @abalkin, @tiran, @Fak3, @JimJJewett, @zooba, @MojoVampire, @ndjensen, @yan12125, @ammaraskar, @cschramm, @pganssle
Files	issue27400.patch 27400.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = None
created_at = <Date 2016-06-27.13:49:18.523>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Datetime NoneType after calling Py_Finalize and Py_Initialize'
updated_at = <Date 2018-07-05.17:14:18.266>
user = 'https://bugs.python.org/DennyWeinberg'

bugs.python.org fields:

activity = <Date 2018-07-05.17:14:18.266>
actor = 'pablogsal'
assignee = 'belopolsky'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2016-06-27.13:49:18.523>
creator = 'Denny Weinberg'
dependencies = []
files = ['44678', '46478']
hgrepos = []
issue_num = 27400
keywords = ['patch']
message_count = 15.0
messages = ['269379', '269381', '269751', '269914', '269916', '276598', '276604', '276622', '276626', '276630', '276632', '277184', '285560', '286618', '286636']
nosy_count = 15.0
nosy_names = ['ncoghlan', 'belopolsky', 'christian.heimes', 'palm.kevin', 'Roman.Evstifeev', 'Jim.Jewett', 'steve.dower', 'josh.r', 'ndjensen', 'yan12125', 'Denny Weinberg', 'ammar2', 'cschramm', 'p-ganssle', 'FrankBlabu']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27400'
versions = ['Python 3.6', 'Python 3.7']

Linked PRs

• gh-71587: Clear cached strptime module #101783
• gh-71587: Isolate _datetime #102995
• gh-71587: Establish global state in _datetime #110475
• gh-71587: Drop local reference cache to _strptime module in _datetime #120224
• [3.13] gh-71587: Drop local reference cache to _strptime module in _datetime (gh-120224) #120424
• [3.12] gh-71587: Drop local reference cache to _strptime module in _datetime (GH-120224) #120431

[DennyWeinberg]

After calling Py_Finalize and Py_Initialize I get the message "attribute of type 'NoneType' is not callable" on the datetime.strptime method.

Example:

from datetime import datetime
s = '20160505 160000'
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')

The first call works fine but it crashes after the re initialization.

Workaround:

from datetime import datetime
s = '20160505 160000'
try:
    refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')
except TypeError:
    import time
    refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d %H%M%S')))

Related Issue: bpo-17408 ("second python execution fails when embedding")

[DennyWeinberg]

Just to be clear:

The error happens after these steps:

1. Call strptime
2. Call cpython function "Py_Finalize" and "Py_Initialize"
3. Call strptime again

Now we get the error "attribute of type 'NoneType' is not callable"

[ncoghlan]

Thanks for the report Denny. Looking at https://hg.python.org/cpython/file/30099abdb3a4/Modules/datetimemodule.c#l3929, there's a problematic caching of the "_strptime" module that is almost certainly the cause of the problem - it will attempt to call _strptime._strptime from the already finalized interpreter rather than the new one.

It should be possible to adjust that logic to permit a check for _strptime._strptime being set to None, and reimporting _strptime in that case.

[ammaraskar]

Is there any particular reason that datetime.strptime caches the imported module like that?

From a quick search, these two other examples don't bother with any caching:

cpython/Modules/timemodule.c

Line 709 in 2d26423

PyObject *strptime_module = PyImport_ImportModuleNoBlock("_strptime");

cpython/Python/traceback.c

Line 277 in 64fe35c

io = PyImport_ImportModuleNoBlock("io");

[ncoghlan]

Aye, skipping the caching entirely would be an even simpler solution - the only thing it is saving in the typical case is a dictionary lookup in the modules cache.

[MojoVampire]

Nick: Looks like it's quite a bit more work than just a dict lookup. That PyImport_ImportModuleNoBlock call (which seems odd; the implementation of NoBlock is just to wrap the blocking function; guess we don't allow non-blocking imports anymore and this is just to avoid changing all the names elsewhere?) involves a *lot* more work than just a dict lookup (it devolves to a PyImport_Import call https://hg.python.org/cpython/file/3.5/Python/import.c#l1743 , which basically does everything involved in the import process aside from actually reading/parsing the file unconditionally, because of how weird __import__ overrides can be, I guess).

While it's not a perfect comparison, compare:

>>> import _strptime  # It's now cached

# Cache globals dict for fair comparison without globals() call overhead
>>> g = globals()     

# Reimport (this might be *more* expensive at C layer, see notes below)
>>> %timeit -r5 import _strptime
1000000 loops, best of 5: 351 ns per loop

# Dict lookup (should be at least a bit cheaper at C layer if done equivalently, using GetAttrId to avoid temporary str)
>>> %timeit -r5 g['_strptime']
10000000 loops, best of 5: 33.1 ns per loop

# Cached reference (should be *much* cheaper at C layer)
>>> %timeit -r5 _strptime
100000000 loops, best of 5: 19.1 ns per loop

Note: I'm a little unclear on whether a Python function implemented in C has its own globals, or whether it's simulated as part of the C module initialization); if it lacks globals, then the work done for PyImport_Import looks like it roughly doubles (it has to do all sorts of work to simulate globals and the like), so that 351 ns per re-import might actually be costlier in C.

Either way, it's a >10x increase in cost to reimport compared to a dict lookup, and ~18x speedup over using a cached reference (and like I said, I think the real cost of the cheaper options would be much less in C, so the multiplier is higher). Admittedly, in tests, empty string calls to _strptime._strptime take around 7.4 microseconds (with realistic calls taking 8.5-13.5 microseconds), so caching is saving maybe a third of a microsecond overhead, maybe 2.5%-4.5% of the work involved in the strptime call.

[MojoVampire]

Hmm... On checking down some of the code paths and realizing there were some issues in 3.5 (redundant code, and what looked like two memory leaks), I checked tip (to avoid opening bugs on stale code), and discovered that bpo-22557 rewrote the import code, reducing the cost of top level reimport by ~60%, so my microbenchmarks (run on Python 3.5.0) are already out of date for 3.6's faster re-import. Even so, caching wasn't a wholly unreasonable optimization before now, and undoing it now still has a cost, if a smaller one.

[abalkin]

I am not sure this is possible to fix without refactoring the datetime module according to PEP-3121. See bpo-15390.

[tiran]

PEP-3121 is a big change. Can we use PyModuleDef->m_clear() for a clever hack?

[abalkin]

Yes, I think something like the attached patch may do the trick.

[tiran]

Wouldn't it clear strptime_module when a subinterpreter shuts down, too? It's not a big deal because it can't cause a crash.