--color does not work under Emacs

[xuchunyang]

I'm using rg 0.2.1 and Emacs 25. It looks like --color doesn't have effect under Emacs, while other tools such as grep works.

• M-! echo hello | grep --color=always hello gives [01;31m�[Khello�[m�[K
• M-! echo hello | rg --color=always hello gives hello (where is the escape code?)

see also emacs-helm/helm#1624 (comment)

[BurntSushi]

Tracking down why colors aren't working has historically been a very challenging task because a lot of it is dependent on your environment. For example:

$ TERM=foobar rg --color=always PATTERN --debug

Colors no longer work. The --debug flag might show a lot of other junk, but in my case, I do see this:

DEBUG:rg::out: error loading terminfo for coloring: could not find a terminfo entry for this terminal

If I try this with GNU grep, colors still work, even if my TERM setting is nonsensical:

TERM=foobar grep --color=always PATTERN *

So this could be one possible explanation. There may be other problems. For example, if TERM is unset:

$ unset TERM
$ rg --color=always PATTERN --debug

In the debug output, I see:

DEBUG:rg::out: error loading terminfo for coloring: TERM environment variable unset, unable to detect a terminal

Once again, GNU grep still works. It seems GNU grep will fall back to some default settings that ripgrep doesn't do, and maybe we should do that. In the mean time, my strong suspicion is that something about your environment isn't configured correctly and GNU grep is bailing you out.

So, can you run rg with --debug and see if you can find the relevant error message? If not, you can just post the entire output. If that doesn't turn up anything, what is the value of TERM when rg is run? What about TERMINFO?

Thanks!

[xuchunyang]

M-! echo hello | rg --debug --color=always hello returns

DEBUG:grep::search: regex ast:
Literal {
    chars: [
        'h',
        'e',
        'l',
        'l',
        'o'
    ],
    casei: false
}
DEBUG:grep::literals: literal prefixes detected: Literals { lits: [Complete(hello)], limit_size: 250, limit_class: 10 }
DEBUG:rg::out: environment doesn't support coloring
hello

M-! echo $TERM returns dumb and TERMINFO is empty, probably not set.

[BurntSushi]

Having TERMINFO unset is OK. ripgrep will look in standard locations for your terminfo database.

Indeed, if I set TERM=dumb, then I also get the same error message as you.

So... It sounds like you need to change your TERM to a setting that supports colors. There's more info on the dumb setting here and here. Running infocmp dumb shows that a dumb terminal has very limited capabilities. If Emacs does indeed support colors, then it sounds like you'll need to change your TERM environment variable to something else.

[thierryvolpiatto]

Actually colors are working fine not only for GNU/grep, but also ag, pt, ack-grep and git-grep, so perhaps the question is: What is different in ripgrep than on e.g ag that make ripgrep failing with a DUMB terminal ?

[BurntSushi]

At least GNU grep hard codes ANSI color escape sequences and AFAIK doesn't actually respect the TERMINFO database. e.g.,

static const char *selected_match_color = "01;31";  /* bold red */

ripgrep isn't failing as far as I can see, it's simply respecting what the environment is saying, while all of the other tools aren't.

What is the correct behavior here? If TERM=dumb is set and --color=always is set, then how does ripgrep know which TERM settings to use? Should it forcefully pick a TERM other than dumb? Which one?

[BurntSushi]

@ticki You seem to know a thing or two about this sort of thing. Can you make sense of this? I notice in your termion library, you don't read from the TERMINFO database and hardcode escape sequences. Does this work correctly across all terminals? (I suppose you're in good company, since GNU grep does the same.) If so, what's the purpose of the TERMINFO database in the first place? For things more advanced than colors?

[thierryvolpiatto]

Andrew Gallant notifications@github.com writes:

At least GNU grep hard codes ANSI color escape sequences and AFAIK doesn't actually respect the TERMINFO database. e.g.,

static const char selected_match_color = "01;31"; / bold red */

ripgrep isn't failing as far as I can see, it's simply respecting what the environment is saying, while all of the other tools aren't.

What is the correct behavior here? If TERM=dumb is set and
--color=always is set, then how does ripgrep know which TERM settings
to use? Should it forcefully pick a TERM other than dumb? Which one?

Ok, just found that if I set TERM to eterm-color colors with ripgrep are
working, so perhaps it is the one to use when TERM=dumb ?

BTW ripgrep is using a basic red color, how do we configure which color
to use ?
Does ripgrep obey to GREP_COLORS env var ? Seems no.
Does it have a config file where we can set this ?

Thierry

[BurntSushi]

Ok, just found that if I set TERM to eterm-color colors with ripgrep are
working, so perhaps it is the one to use when TERM=dumb ?

"it works for me" isn't really a strong reason to assume "it always works," unfortunately.

BTW ripgrep is using a basic red color, how do we configure which color
to use ?
Does ripgrep obey to GREP_COLORS env var ? Seems no.
Does it have a config file where we can set this ?

Colors aren't yet configurable. See: #51

[BurntSushi]

From reading the source of termion, I wonder if it'd be better to stop using the term crate, switch to termion for POSIX compatible stuff and handroll Windows support. This would mean not supporting the TERMINFO database, and I'm not sure what the implications of that are. (But it would also mean that colors would work independently of a TERMINFO database.)

[BurntSushi]

@thierryvolpiatto @xuchunyang I think the bottom line here is that while ripgrep is technically respecting the letter of the law, it doesn't actually match up with user expectations and that's bad. That means I agree we should do something here, but the right answer might take a bit of time to happen. For now, I'd request setting TERM to something that makes ripgrep work as a temporary work-around. Hopefully that's feasible to do.

[thierryvolpiatto]

Andrew Gallant notifications@github.com writes:

@thierryvolpiatto @xuchunyang I think the bottom line here is that
while ripgrep is technically respecting the letter of the law, it
doesn't actually match up with user expectations and that's bad. That
means I agree we should do something here, but the right answer might
take a bit of time to happen.

There is no hurry, thanks for all your explanations and sorry to not be
able to help on this, my knowledge about all this is limited.

For now, I'd request setting TERM to something that makes ripgrep work
as a temporary work-around. Hopefully that's feasible to do.

Ok, we will use TERM=eterm-color as a workaround for now.

Thanks again for your help on this.

Thierry

[ticki]

@BurntSushi So TERMINFO is mostly for detecting compatibility, but is only really relevant for very old terminals. Termion implements escapes which are supported by virtually every VTE used today (most of it is straight out of the ANSI standards, some are ones introduced by xterm). For the Windows stuff: I'm not that familiar with the Windows API, but I'm more than willing to accept a PR adding Windows support to Termion (i.e. adding the winapi calls to the formatter, I guess).

[BurntSushi]

@ticki Got ya. Given that every other tool in this space seems to not care about TERMINFO, it seems OK for ripgrep to do the same. With that in mind, I'd like to switch Unix handling to using termion, which I think should be straight-forward.

I would definitely recommend treading carefully with Windows. For Windows, you need to make synchronous API calls to the console you're writing to, which means that you can't really implement Windows support in some arbitrary formatter that can write to anything. For ripgrep, this is particularly bad because we want to write matches to an in memory buffer and only actually print them later. There are a couple ways of doing this, including writing an ANSI interpreter, but I opted for something simpler but just as hacky: https://github.com/BurntSushi/ripgrep/blob/master/src/terminal_win.rs

What I'm saying is: as a future user of termion, I don't expect you to support Windows for this. I expect to be subjected to grueling pain. :-)