Multiline lookbehind, empty submatches array

[roblourens]

If I search with a lookbehind pattern that matches a newline, and use the --json flag, I get the match info but it has an empty submatches array. Example:

$ echo -e 'foo\nbar' > test.txt
$ rg --auto-hybrid-regex --multiline --json '(?<=foo\n)bar' test.txt
{"type":"begin","data":{"path":{"text":"test.txt"}}}
{"type":"match","data":{"path":{"text":"test.txt"},"lines":{"text":"bar\n"},"line_number":2,"absolute_offset":4,"submatches":[]}}
{"type":"end","data":{"path":{"text":"test.txt"},"binary_offset":null,"stats":{"elapsed":{"secs":0,"nanos":40700,"human":"0.000041s"},"searches":1,"searches_with_match":1,"bytes_searched":8,"bytes_printed":183,"matched_lines":1,"matches":0}}}
{"data":{"elapsed_total":{"human":"0.002328s","nanos":2328499,"secs":0},"stats":{"bytes_printed":183,"bytes_searched":8,"elapsed":{"human":"0.000041s","nanos":40700,"secs":0},"matched_lines":1,"matches":0,"searches":1,"searches_with_match":1}},"type":"summary"}

Compare to what I get when matching with lookbehind on one line:

$ rg --auto-hybrid-regex --multiline --json '(?<=b)ar' test.txt 
...
{"type":"match","data":{"path":{"text":"test.txt"},"lines":{"text":"bar\n"},"line_number":2,"absolute_offset":4,"submatches":[{"match":{"text":"ar"},"start":1,"end":3}]}}

Or just matching that line

$ rg --auto-hybrid-regex --multiline --json 'bar' test.txt
...
{"type":"match","data":{"path":{"text":"test.txt"},"lines":{"text":"bar\n"},"line_number":2,"absolute_offset":4,"submatches":[{"match":{"text":"bar"},"start":0,"end":3}]}}

Same with lookahead that matches a newline.

[alessandroasm]

Hi, I looked into what is happening in this case.

ripgrep/crates/printer/src/json.rs

Lines 604 to 614 in def993b

	/// Execute the matcher over the given bytes and record the match
	/// locations if the current configuration demands match granularity.
	fn record_matches(&mut self, bytes: &[u8]) -> io::Result<()> {
	self.json.matches.clear();
	// If printing requires knowing the location of each individual match,
	// then compute and stored those right now for use later. While this
	// adds an extra copy for storing the matches, we do amortize the
	// allocation for it and this greatly simplifies the printing logic to
	// the extent that it's easy to ensure that we never do more than
	// one search to find the matches.
	let matches = &mut self.json.matches;

The issue is that grep_searcher::SinkMatch doesn't store captured groups. When JSONSink needs details about capture groups, it reruns the matcher against the bytes in SinkMatch, but those bytes only contain the current line, which in turn results in a non-match scenario.

I think the way to go here will be including submatch data in grep_searcher::SinkMatch. That way we won't need to rerun the matcher to get this information.

@BurntSushi I can implement that if you agree to this changes