unittest: execute tests in parallel

[donpellegrino]

BPO	37873
Nosy	@terryjreedy, @giampaolo, @ezio-melotti, @voidspace, @bharel, @tirkarthi, @donpellegrino

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-08-16.14:15:27.971>
labels = ['type-feature', 'library', '3.9']
title = 'unittest: execute tests in parallel'
updated_at = <Date 2020-09-02.09:57:36.866>
user = 'https://github.com/donpellegrino'

bugs.python.org fields:

activity = <Date 2020-09-02.09:57:36.866>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-08-16.14:15:27.971>
creator = 'user93448'
dependencies = []
files = []
hgrepos = []
issue_num = 37873
keywords = []
message_count = 5.0
messages = ['349864', '349938', '350080', '375126', '375128']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'giampaolo.rodola', 'ezio.melotti', 'michael.foord', 'bar.harel', 'xtreak', 'user93448']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37873'
versions = ['Python 3.9']

Linked PRs

• gh-82054: Implements test.regrtest unittest sharding to pull in long tails #99637
• gh-82054: allow test runner to split test_asyncio #103859
• gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. #103927
• gh-82054: Parallel test sharding of test_asyncio and test_multiprocessing #103934

[donpellegrino]

The unittest documentation makes reference to a potential parallelization feature:

"Note that shared fixtures do not play well with [potential] features like test parallelization and they break test isolation. They should be used with care." (https://docs.python.org/3/library/unittest.html)

However, it seems that executing tests in parallel is not yet a feature of unittest. This enhancement request is to add parallel execution of tests to unittest.

A command line option may be a good interface. Ideally, it would be compatible with test discovery. Outside of the Python ecosystem, a common practice is to define test cases in a Makefile and then execute GNU Make with the '-j' flag (https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel). Adding such an option to unittest would be a convenience and may save the effort of bringing in additional libraries or tools for parallel unit test execution.

[terryjreedy]

test.regrtest has a -j option. Perhaps some of the Python coding for that could be used for unitest also.

[tirkarthi]

See also https://mail.python.org/pipermail/python-ideas/2017-September/047100.html . One of the ideas in the thread was to move test.regrtest parallel execution functionality into unittest. I think this would be good to have it in unittest like support in pytest for -j.

[donpellegrino]

Leveraging GNU Parallel (https://www.gnu.org/software/parallel/) might help simplify implementation. Perhaps that could be used as a subprocess call?

[vstinner]

Leveraging GNU Parallel (https://www.gnu.org/software/parallel/) might help simplify implementation. Perhaps that could be used as a subprocess call?

In general, we attempt to avoid depending on the availability of external tool. For example, I don't expect this tool to be available on Windows, whereas it would be better to support parallel execution on Windows as well.

[gpshead]

An example of implementing sharding within a single unittest so they can run in parallel across whatever number of processes and machines you want is what we use at work: https://github.com/abseil/abseil-py/blob/v1.3.0/absl/testing/absltest.py#L2359 using the Bazel sharding protocol.

If test.regrtest supported using such a sharding protocol and unittest adopted that TestCase.getTestCaseNames implementation logic to compute the shards for each process, we'd really pull in our long tail test time in CPython's own unittest suite.

One of the ideas in the thread was to move test.regrtest parallel execution functionality into unittest. I think this would be good to have it in unittest like support in pytest for -j.

Yes. Though wider Python community wise you'll just find many people saying "just use pytest-xdist" in your project for -j support. I can't disagree with that.

I still want parallelization for CPython's own unittest suite (launched via python -m test) regardless of that. It'd help save core dev & contributor time as well as increasing buildbot and CI thruput.