timeit setup is affected by test statement

[jaraco]

Bug description:

In #114664, I've stumbled on an unexpected behavior in the timeit module.

According to the docs, the "setup" function is run once before repeating the statement under test, but that's not what I'm observing:

 ~ @ py -m timeit --setup 'import sys; assert "csv" not in sys.modules' 'import csv'
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 330, in main
    number, _ = t.autorange(callback)
                ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 226, in autorange
    time_taken = self.timeit(number)
                 ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 180, in timeit
    timing = self.inner(it, self.timer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<timeit-src>", line 3, in inner
    import sys; assert "csv" not in sys.modules
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Compared with

 ~ @ py -m timeit --setup 'import sys; assert "csv" not in sys.modules' pass
100000000 loops, best of 5: 3.89 nsec per loop

In the first, the assertion fails in the setup. But why should it fail? If it's run before the import csv statement, then the assertion should pass. How is it that "csv" is in sys.modules during setup?

CPython versions tested on:

3.12

Operating systems tested on:

macOS

[hauntsaninja]

I responded with specifics about what timeit is doing here in #114664 (comment) , but maybe the documentation could be improved

[AlexWaygood]

FWIW @jaraco, this is a script I have hanging around locally for measuring import times. It avoids the pitfalls you've correctly identified with using timeit for this task. You might find it helpful :) (Or you can use an external tool like hyperfine, as @hauntsaninja did in #114664 (comment))

If you save it as bench_import.py, you execute it with e.g. python bench_import.py typing from the command line to measure the import time of typing.

import sys
import time
import statistics
import subprocess

module = sys.argv[1]
times = []

for _ in range(500):
    ret = subprocess.run(
        [sys.executable, "-c", f"import time; t0 = time.perf_counter(); import {module}; print(time.perf_counter() - t0)"],
        check=True,
        capture_output=True,
        text=True
    )
    times.append(float(ret.stdout.strip()))

print(statistics.mean(times))

Note that Python also has a special mode builtin for profiling import time: you can run python -X importtime "import typing" to create a log file that shows exactly what's making a module slow to import. https://docs.python.org/3/using/cmdline.html#cmdoption-X