Add more granularity to _pyrepl syntax theming for "keywords"

[treyhunner]

Feature or enhancement

Proposal:

I made a custom _colorize theme to try making my REPL syntax highlighting look similar to my text editor's.

This is the theme I used:

theme_with_dim_operators = default_theme.copy_with(
    syntax=Syntax(
        keyword=ANSIColors.GREEN,
        builtin=ANSIColors.BLUE,
        comment=ANSIColors.INTENSE_YELLOW,
        string=ANSIColors.CYAN,
        number=ANSIColors.CYAN,
        definition=ANSIColors.BLUE,
        soft_keyword=ANSIColors.BOLD_GREEN,
    ),
)

I then used this example code to check the highlighting between the two:

from math import pi
print(f"Welcome to Python {pi:.2f}")
name = None
if name is None:
    print("It's", True)

Pasting that code in my text editor (vim), looks like this:

Pasting the same code in the REPL, looks like this:

The granularity with syntax highlighting needs to stop somewhere, but I feel it would be nice to add a bit more granularity than is currently supported by _colorize.

In particular, it would be nice to distinguish between:

1. Syntax keywords in general (if, is, def, for, with, etc.)
2. Keywords that are variables (True, False, None)
3. Imports in particular (from, import)

I feel that distinguishing between 1 and 2 is more important than between 1 and 3. While None is a keyword, seeing it appear in the same color as is and other keywords feels odd.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-134953: Expand theming for True/False/None #135000
• [3.14] gh-134953: Expand theming for True/False/None (GH-135000) #138928
• gh-134953: Make the True/False/None check more efficient #138931
• [3.14] gh-134953: Make the True/False/None check more efficient (GH-138931) #138939

[treyhunner]

I looked into how Pygments highlights and how MagicPython (used by VS Code) highlights. Both have much more granularity than I expect _colorize will ever support, but I do think it's worth giving _colorize slightly more granularity in syntax highlighting.

Pygments uses:

• Keyword.Constant for True, False, and None
• Keyword.Namespace for import and from
• Keyword for if and most other keywords
• Operator.Word for is, and, or, and not

It seems like MagicPython uses:

• constant.language.python for True, False, and None
• keyword.control.import.python for import and from
• keyword.control.import.python for if and most other keywords
• keyword.operator.logical.python for is, and, or, and not

Both of these syntax highlighters also seem to support a hierarchy (if custom highlighting is not defined for and/or/is then the same highlighting used for if and other keywords will be used.

[ambv]

The way we syntax color currently only tokenizes the code, it doesn't parse the code (modulo soft keywords). Therefore, differentiating "as" and "from" in imports is currently impossible. I would leave it out.

I did land True/False/None differentiation, that makes sense since those keywords are values.