Library multiprocess leaks named resources.

[jxdabc]

BPO	46391
Nosy	@pitrou, @benjaminp, @1st1, @applio, @arhadthedev, @jxdabc, @BarkingBad
PRs	gh-90549: Fix leak of global named resources using multiprocessing spawn #30617
Files	screen.png: screenshot

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-01-15.18:04:57.278>
labels = ['3.8', '3.9', '3.10', 'performance', '3.11', 'library']
title = 'Library multiprocess leaks named resources.'
updated_at = <Date 2022-02-18.19:39:09.294>
user = 'https://github.com/jxdabc'

bugs.python.org fields:

activity = <Date 2022-02-18.19:39:09.294>
actor = 'BarkingBad'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-01-15.18:04:57.278>
creator = 'milestonejxd'
dependencies = []
files = ['50580']
hgrepos = []
issue_num = 46391
keywords = ['patch']
message_count = 5.0
messages = ['410654', '410657', '410853', '411456', '413504']
nosy_count = 8.0
nosy_names = ['pitrou', 'benjamin.peterson', 'sbt', 'yselivanov', 'davin', 'arhadthedev', 'milestonejxd', 'BarkingBad']
pr_nums = ['30617']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46391'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[jxdabc]

Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python.

Many Repo users have encountered resource leak warnings with Python 3.9.
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5

I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess.

To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. )

import multiprocessing as mp

global_resource = mp.Semaphore()

def submain(): pass

if __name__ == '__main__':
    p = mp.Process(target=submain)
    p.start()
    p.join()

Tested on macOS with Python 3.9.7

python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown.

This bug will 100% reproduce when then main module uses named resources as global variables and uses spawn context, which is the case of Repo on macOS.

This is caused by multiprocess::BaseProcess::_bootstrap.

When a new process is started with multiprocessing.Process.start() in spawn context.

1. The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main.
2. Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves.
3. multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.

When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else.

And I uploaded a MR.

[arhadthedev]

I added core devs related to multiprocessing into a nosy list so they got a notification and the PR will be evaluated and merged faster. FYI, the devs are Davin Potts and Antoine Pitrou (as per <https://devguide.python.org/experts/\>).

[jxdabc]

loop file owner.

[jxdabc]

update,

Confirmed to affect all versions above 3.8+.

It's really annoying.

Command line tools like Repo get a warning after every (success) command.
And mess up with shell.

-------------------------------------------------------------------

$ repo status .
project ksbox/                                  branch dev
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo sync .
Fetching: 100% (1/1), done in 1.184s
Garbage collecting: 100% (1/1), done in 0.013s
repo sync has finished successfully.
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo status .
project ksbox/ branch dev
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '