concurrent.futures.wait returns cancelled as "not done"

[cweberprosperitybh]

Bug report

Bug description:

According to the documentation for concurrent.futures.wait, it:

Returns a named 2-tuple of sets. The first set, named done, contains the futures that completed (finished or cancelled futures) before the wait completed. The second set, named not_done, contains the futures that did not complete (pending or running futures).

In this example, however, it returns cancelled futures in not_done. Note the wait=True in the preceding executor.shutdown call, which should ensure futures are not cancelled between wait and print. task calls sleep to ensure at least one thread is pending when executor.shutdown is called, even on especially performant systems.

from concurrent.futures import ThreadPoolExecutor, wait
from time import sleep


def task():
    sleep(1)


executor = ThreadPoolExecutor(max_workers=1)
futures = [executor.submit(task), executor.submit(task)]
executor.shutdown(wait=True, cancel_futures=True)
wait_return = wait(futures, timeout=0)
print(wait_return.not_done)

Considerations

• If we instead call wait with timeout=None, the script hangs indefinitely.
• If we replace the call to executor.shutdown with futures[1].cancel() (leave timeout=0), the cancelled future is similarly included in not_done. We can add print(futures[1].cancelled()) to ensure the cancellation is successful before the call to wait.
• Replacing both, however, produces the expected result.

I couldn't find an existing report; apologies if I missed it.

CPython versions tested on:

3.11

Operating systems tested on:

Linux, macOS

Linked PRs

• gh-109934: notify cancelled futures on thread pool shutdown #134618

[duaneg]

This appears to be a bug in concurrent.futures.ThreadPoolExecutor: when the pool is shutdown it cancels all pending work items, however it does not call set_running_or_notify_cancel on the future, so it stays in the CANCELLED state instead of being moved to the CANCELLED_AND_NOTIFIED state.

Aside from the task being incorrectly reported as "not done", this means code that is blocking waiting on the future to complete will never be awakened (e.g. if passing wait=True to shutdown, as the OP notes, or if using as_completed etc).