[WASI] test cases with sys.path.insert(0, os.curdir) fail on WASI

[tiran]

Bug report

Test cases that insert os.curdir ('.') into sys.path fail on WASI platforms. I think it is related how wasmtime maps directories into the runtime environment and how it sets up the current working directory. The test passes if I replace '.' with an empty string '' or remove the cached FileFinder instance for '.' with sys.path_importer_cache.pop('.').

 ./test-wasi.sh test_support -m test_forget
0:00:00 Run tests sequentially
0:00:00 [1/1] test_support
test test_support failed -- Traceback (most recent call last):
  File "/Lib/test/test_support.py", line 116, in test_forget
    mod = __import__(TESTFN)
          ^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '@test_66600_tmpæ'

test_support failed (1 error)

== Tests result: FAILURE ==

The culprit seems to be an outdated FileFinder instance: {'.': FileFinder('/.')}. The CWD is '/python-wasm/cpython/builddir/wasi/build/test_python_66600æ' when the test case is executed.

0:00:00 [1/1] test_support
> /Lib/test/test_support.py(114)test_forget()
-> sys.path.insert(0, os.curdir)
(Pdb) p sys.path
['.', '/usr/local/lib/python312.zip', '/Lib', '/build/lib.wasi-wasm32-3.12']
(Pdb) pp sys.path_importer_cache
{'.': FileFinder('/.'),
 '/Lib': FileFinder('/Lib'),
 '/Lib/collections': FileFinder('/Lib/collections'),
 '/Lib/encodings': FileFinder('/Lib/encodings'),
 '/Lib/importlib': FileFinder('/Lib/importlib'),
 '/Lib/re': FileFinder('/Lib/re'),
 '/Lib/test': FileFinder('/Lib/test'),
 '/Lib/test/libregrtest': FileFinder('/Lib/test/libregrtest'),
 '/Lib/test/support': FileFinder('/Lib/test/support'),
 '/Lib/unittest': FileFinder('/Lib/unittest'),
 '/Lib/urllib': FileFinder('/Lib/urllib'),
 '/build/lib.wasi-wasm32-3.12': FileFinder('/build/lib.wasi-wasm32-3.12'),
 '/usr/local/lib/python312.zip': None}
(Pdb) p os.getcwd()
'/python-wasm/cpython/builddir/wasi/build/test_python_66600æ'
(Pdb) n
> /Lib/test/test_support.py(115)test_forget()
-> importlib.invalidate_caches()
(Pdb) n
> /Lib/test/test_support.py(116)test_forget()
-> try:
(Pdb) p sys.path
['.', '.', '/usr/local/lib/python312.zip', '/Lib', '/build/lib.wasi-wasm32-3.12']
(Pdb) pp sys.path_importer_cache
{'.': FileFinder('/.'),
 '/Lib': FileFinder('/Lib'),
 '/Lib/collections': FileFinder('/Lib/collections'),
 '/Lib/encodings': FileFinder('/Lib/encodings'),
 '/Lib/importlib': FileFinder('/Lib/importlib'),
 '/Lib/re': FileFinder('/Lib/re'),
 '/Lib/test': FileFinder('/Lib/test'),
 '/Lib/test/libregrtest': FileFinder('/Lib/test/libregrtest'),
 '/Lib/test/support': FileFinder('/Lib/test/support'),
 '/Lib/unittest': FileFinder('/Lib/unittest'),
 '/Lib/urllib': FileFinder('/Lib/urllib'),
 '/build/lib.wasi-wasm32-3.12': FileFinder('/build/lib.wasi-wasm32-3.12'),
 '/usr/local/lib/python312.zip': None}

A build on Linux has an absolute path to build directory as first sys.path entry.

reproducer

$ podman run --rm -ti -v $(pwd):/python-wasm/cpython:Z quay.io/tiran/cpythonbuild:emsdk3
# ./build-python-build.sh
# ./build-python-wasi.sh
# ./test-wasi.sh test_support -m test_forget

[tiran]

GH-93461 is the simplest solution I could come up with. importlib.invalidate_caches now drops all FileFinder instances if the cache key is not an absolute path.

[brettcannon]

Seems like a reasonable solution to the problem.

[brettcannon]

@tiran can this be closed?

[tiran]

Yes, it is fixed now.