Improve PyREPL auto-indentation

[tanloong]

Bug report

On second thought this issue should be an enhancement instead of a bug report. Sorry for the wrong template.

Bug description:

There are some cases where _pyrepl auto-indentation works not well.

Cases

1. A line ending with : in a multi-line string is wrongly indented.

Observed

>>> s = '''
... Note:
... ␣␣␣␣|

Expected

>>> s = '''
... Note:
... |

2. # inside strings is seen as a comment, the following : is ignored.

Observed

>>> if ' ' == '#':
... |

Expected

>>> if ' ' == '#':
... ␣␣␣␣|

3. When the entire cursor line is a comment and is already indented, pressing Enter gives a further indent.

Observed

>>> def f():
...     # foo⤶
... ␣␣␣␣␣␣␣␣|

Expected

>>> def f():
...     # foo⤶
... ␣␣␣␣|

Possible solution

Currently _should_auto_indent() parses the buffer from right to left and stops at the first newline it encounters. Only the last line that is not a comment line of the buffer is parsed.

But by parsing from right to left we can't tell if a # starts a comment or is part of a string. For example if we, from right to left, encounter a " first and then a #, we don't know if the # is a comment. To know that, we need to know if the " is a string boundary, but the # might comment out the ", so we can't be sure. There is a information dependency cycle.

To fix this I made a change to parse the buffer from left to right, keeping track of whether current char is inside a string or a comment. This approach solves the above three cases. However the whole buffer is parsed on every call of _should_auto_indent(), with very long buffer, there might be noticeable delay when pressing Enter.

I think this is a big change. It affects how _should_auto_indent() works as a whole. I am hesitated to create a PR and just put it here first to hopefully get feedback.

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Linked PRs

• gh-133710: Enhance pyrepl auto-indent #140710

[pablogsal]

However the whole buffer is parsed on every call of _should_auto_indent(), with very long buffer, there might be noticeable delay when pressing Enter.

I think that this may unfortunately be a deal breaker. Thoughts @ambv ?

[tanloong]

Thanks for the response! I will see if I can profile it over the weekend. If it ends up causing significant lag, what do you think about left-to-right full parsing only when the buffer is under certain threshold length, and falling back to the original right-to-left partial parsing for longer buffer?

[ambv]

I would prefer to keep the implementation simple and not split behavior based on input size. What problem are we solving here exactly? Just that auto-indent works inside multiline strings?

[tanloong]

@ambv Apologies for the split behavior suggestion, I agree it would complicate the implementation unnecessarily.

@pablogsal I've profiled the performance impact of the change. Here's a comparison between the original and enhanced versions when processing a multiline string with up to 10k characters. While the relative slowdown is significant, the absolute delay remains low (~0.02s for 10k characters on an i5-7200U 2.5GHz CPU with an HDD). The delay is not noticeable, I think, and in interactive REPL scenarios the inputs would be shorter.

The code:

import timeit
import random
import string

def _should_auto_indent__main_branch(buffer: list[str], pos: int) -> bool:
    # check if last character before "pos" is a colon, ignoring
    # whitespaces and comments.
    last_char = None
    while pos > 0:
        pos -= 1
        if last_char is None:
            if buffer[pos] not in " \t\n#":  # ignore whitespaces and comments
                last_char = buffer[pos]
        else:
            # even if we found a non-whitespace character before
            # original pos, we keep going back until newline is reached
            # to make sure we ignore comments
            if buffer[pos] == "\n":
                break
            if buffer[pos] == "#":
                last_char = None
    return last_char == ":"

def _should_auto_indent__enhance_branch(buffer: list[str], pos: int) -> bool:
    # check if last character before "pos" is a colon, ignoring
    # whitespaces, comments, and contents within strings.
    last_char = None
    # A stack to keep track of string delimiters. Push a quote when entering a
    # string, pop it when the string ends. If the stack is empty, we're not
    # inside a string. When see a '#', it's a comment start if we're not inside
    # a string; otherwise, it's just a '#' character within a string.
    str_delims: list[str] = []
    in_comment = False
    char_line_indent_start: int | None = None
    char_line_indent = 0
    lastchar_line_indent = 0
    cursor_line_indent = 0

    i = -1
    while i < pos - 1:
        i += 1
        char = buffer[i]

        # update last_char
        if char == "#":
            if str_delims:
                last_char = char # '#' inside a string is just a character
            else:
                in_comment = True
        elif char == "\n":
            # newline ends a comment
            in_comment = False
            if i < pos - 1 and buffer[i + 1] in " \t":
                char_line_indent_start = i + 1
            else:
                char_line_indent_start = None # clear last line's line_indent_start
                char_line_indent = 0
        elif char not in " \t":
            if char_line_indent_start is not None:
                char_line_indent = i - char_line_indent_start
            if not in_comment and not str_delims:
                # update last_char with non-whitespace chars outside comments and strings
                last_char = char
                lastchar_line_indent = char_line_indent

        # update stack
        if char in "\"'" and (i == 0 or buffer[i - 1] != "\\"):
            if str_delims and str_delims[-1] == char:
                str_delims.pop()
            else:
                str_delims.append(char)
    cursor_line_indent = char_line_indent
    return last_char == ":" and cursor_line_indent <= lastchar_line_indent

# 1000x100 multilines string
multiline_str = "\n".join("".join(random.choices(string.ascii_letters, k=100)) for _ in range(1000))

if __name__ == "__main__":
    for i in range(2, 12, 2):
        buffer = list(f'"""\n{multiline_str[:i*1000]}\n"""')
        pos = len(buffer)

        t1 = timeit.timeit("_should_auto_indent__main_branch(buffer, pos)", number=100, globals=globals()) / 100
        t2 = timeit.timeit("_should_auto_indent__enhance_branch(buffer, pos)", number=100, globals=globals()) / 100

        print("Multiline string length:", i*1000)
        print("before:", t1)
        print("after:", t2)
        print("diff:", f"{t2-t1=}, {t2/t1=}", "\n")

The result:

Multiline string length: 2000
before: 7.77800043579191e-06
after: 0.0038612419995479284
diff: t2-t1=0.0038534639991121365, t2/t1=496.43118837840484

Multiline string length: 4000
before: 7.662000134587288e-06
after: 0.007558695999905467
diff: t2-t1=0.00755103399977088, t2/t1=986.5173410510015

Multiline string length: 6000
before: 7.987999706529081e-06
after: 0.011519652000279166
diff: t2-t1=0.011511664000572637, t2/t1=1442.1197325362255

Multiline string length: 8000
before: 9.829999762587248e-06
after: 0.01693329099973198
diff: t2-t1=0.016923460999969393, t2/t1=1722.6135715872238

Multiline string length: 10000
before: 7.742000161670148e-06
after: 0.019101623999886214
diff: t2-t1=0.019093881999724543, t2/t1=2467.2724878586805