Glob behavior is inconsistent across libraries

[jaraco]

the [Python] docs for pathlib.glob are unclear on the meaning of **

You are right, and it gets even more confusing:

• glob.glob('**') returns dirs and files, non-recursively, not including .
• glob.glob('**', recursive=True) returns dirs and files, recursively, not including .
• list(pathlib.Path('.').glob('**')) returns only dirs, recursively, and also includes .

I think there should be a Python upstream issue to document all of that, including the differences between glob and pathlib.

Then zipp could implement what's documented for pathlib.

Repro

Python 3.10.11

mkdir -p dir/subdir ; touch toplevel.txt dir/indir.txt

creates:

toplevel.txt
dir/
  indir.txt
  subdir/

>>> import glob, pathlib

>>> glob.glob('**')
['dir', 'toplevel.txt']

>>> glob.glob('**', recursive=True)
['dir', 'dir/subdir', 'dir/indir.txt', 'toplevel.txt']

>>> list(pathlib.Path('.').glob('**'))
[PosixPath('.'), PosixPath('dir'), PosixPath('dir/subdir')]

Originally posted by @nh2 in jaraco/zipp#102 (comment)

Linked PRs

• GH-106747: Improve Path.glob() expectations in pathlib tests #112365
• GH-106747: Make pathlib ABC globbing more consistent with glob.glob() #115056
• GH-106747: Prepare pathlib globbing for dir_fd support. #116319
• GH-106747: Document another difference between glob and pathlib. #116518

[jaraco]

Without having thought about it too much, I'd argue pathlib.Path.glob('**') should return dirs and files recursively, not including ., same as glob.glob(recursive=True). I'd also argue that glob.glob('**') should imply recursive=True. After all, what's the meaning of ** if not recursive?

[jaraco]

@barneygale Do you have any insight on the concern? Any opinions?

[jaraco]

Another point of reference, ls on Unix also treats ** as recursive without the dot:

# ls **
dir/indir.txt  toplevel.txt

dir:
indir.txt  subdir

dir/subdir:

[barneygale]

Yeah, it's a mess

the [Python] docs for pathlib.glob are unclear on the meaning of **

Logged as #101112, stale-ish PR up at #101223

• glob.glob('**', recursive=True) returns dirs and files, recursively, not including .

Try glob.glob('tests/**', recursive=True) - you'll find that tests is included in the results, and so ** is capable of consuming zero path segments. The fact that it doesn't emit . seems like the bug to me! Also conflicts with the docs.

• list(pathlib.Path('.').glob('**')) returns only dirs, recursively, and also includes .

Logged as #70303, PR up as #105413.

[jaraco]

I'm going to be nitpicky because this issue demands it. Understand my matter of fact tone below may come off as hostile, but it's not intended that way. I'm merely trying to lay out the facts and disentangle the concerns.

you'll find that tests is included in the results

More precisely, tests/ is included in the results:

 cpython main @ glob.glob('Lib/test/**', recursive=True)[0]
'Lib/test/'

Note I'm using xonsh shell, which is why you see python interleaved with a shell prompt in the cpython repo.

Interestingly, tests/ is not included in the results if recursive=False.

The fact that it doesn't emit . seems like the bug to me! Also conflicts with the docs.

That's the opposite of my intuition. It's almost never the case that a user wants . in their globs (any more than they want ..). When I do glob('mydir/*'), I want to find all of the contents of "mydir", and "mydir" itself is not contained in "mydir".

Even pathlib only emits the . when a recursive glob is indicated:

 cpython main @ next(pathlib.Path().glob('*'))
PosixPath('PC')
 cpython main @ next(pathlib.Path().glob('**'))
PosixPath('.')

I'm now confused. These two statements seem contradictory:

tests is included in the results

it doesn't emit .

In the case of glob.glob('tests/**', recursive=True), tests/ is emitted and that's equivalent to tests/.. Are you saying you think it's a bug that tests/ is emitted without a dot?

Also, I'm unsure how it conflicts with the docs. Can you elaborate on where the conflict lies?

[barneygale]

That's the opposite of my intuition. It's almost never the case that a user wants . in their globs (any more than they want ..). When I do glob('mydir/*'), I want to find all of the contents of "mydir", and "mydir" itself is not contained in "mydir".

Even pathlib only emits the . when a recursive glob is indicated:

 cpython main @ next(pathlib.Path().glob('*'))
PosixPath('PC')
 cpython main @ next(pathlib.Path().glob('**'))
PosixPath('.')

I was talking about **, not *. The behaviour of standalone * pattern segments is well-defined: it always consumes one path segment.

In the case of glob.glob('tests/**', recursive=True), tests/ is emitted and that's equivalent to tests/.. Are you saying you think it's a bug that tests/ is emitted without a dot?

I'm saying that these two function calls should return the same result, as they do in pathlib:

>>> def myglob(pattern, root_dir=''):
...     for path in glob.glob(pattern, root_dir=root_dir, recursive=True):
...         yield os.path.join(root_dir, path)
... 
>>> list(myglob('testdir/**', ''))
['testdir/', 'testdir/a', 'testdir/b']
>>> list(myglob('**', 'testdir'))
['testdir/a', 'testdir/b']

Also, I'm unsure how it conflicts with the docs. Can you elaborate on where the conflict lies?

** is described as matching "any files and zero or more directories". You can see that, in the testdir/** example above, it can consume zero files and zero directories. But in the ** example, it doesn't consume the current directory.

[barneygale]

FWIW, there is no single standard for how ** should work. Different shells and programming languages implement it differently. See https://stackoverflow.com/questions/70613723/recursive-globs-or-globstar-different-on-zsh-bash-python-and-ruby

[barneygale]

FYI, I'm looking to add a new sep argument to fnmatch.translate() which would unify its pattern-matching with pathlib.PurePath.match() and pathlib.Path.glob(). In doing so it would make fnmatch.translate() more compatible with glob.glob(), but there would still be differences, such as the above-noted difference in how ** works, but also the default treatment of hidden files/directories. PR here:

• GH-72904: Add glob.translate() function #106703

[jaraco]

I'm very interested in reviewing the proposed change. I'm particularly interested to see if it obviates or conflicts with zipp.glob.