--files performance regression in 0.8.0

[chrmarti]

What version of ripgrep are you using?

0.8.0

What operating system are you using ripgrep on?

Windows 10

If this is a bug, what are the steps to reproduce the behavior?

Using an old commit of the Chromium sources (git clone https://chromium.googlesource.com/chromium/src) as corpus (~230k files).

With 0.8.0 --files runs about 4x longer:

PS C:\devel\testWorkspace> (Measure-Command { & 'C:\devel\ripgrep-0.8.0-x86_64-pc-windows-msvc\rg.exe' --files | Measure-Object –Line | Out-Default }).TotalSeconds

 Lines Words Characters Property
 ----- ----- ---------- --------
234319 [files]

43.309153 [seconds]

Than with 0.7.1:

PS C:\devel\testWorkspace> (Measure-Command { & 'C:\devel\ripgrep-0.7.1-x86_64-pc-windows-msvc\rg.exe' --files | Measure-Object –Line | Out-Default }).TotalSeconds

 Lines Words Characters Property
 ----- ----- ---------- --------
234319 [files]

11.2432563 [seconds]

Using Process Monitor it looks like 0.8.0 accesses individual files a lot more:

Than 0.7.1, which seems to only access folders:

/cc @roblourens

[BurntSushi]

@chrmarti Thanks for another great bug report! I can indeed reproduce this. Your tip about ripgrep stat'ing more files is right on the money. Basically, this is fallout from fixing #705. In particular, in order to fix #705, I had to work-around the corresponding bug in Rust's standard library, and I was apparently careless enough that I introduced an extra stat call for each file that ripgrep visits. This should be possible to fix, but I'll need to take a closer look tomorrow.

Assuming I get this fixed, and after I get some of the PR backlog cleared, I can put out another release.

[BurntSushi]

Note that this regression is also in walkdir, so it impacts both single threaded and multi-threaded directory traversal.

[BurntSushi]

This should be fixed in master. I manually QAed this and confirmed that performance is back to 0.7.1 levels for your specific test case. (Both single and multi threaded.)

If all goes well, I'm going to try and squeak a 0.8.1 release out tonight.