Buffer.toString('utf8') appears to use wtf-8

[AljoschaMeyer]

The byte sequence 237, 166, 164 is not valid utf8, since it encodes a surrogate code point, which is not a valid unicode scalar value. So Buffer.from([237, 166, 164]).toString('utf8') should error. But instead, it returns a string, effectively implementing wtf-8 rather than utf-8.

Or does Buffer.toString simply not provide any validity guarantees at all, returning garbage strings if the buffer contains invalid input? In that case, please document this as expected behavior, since it makes the function completely useless for a bunch of use cases.

node -v: v10.11.0
uname -a: Linux aljoscha-laptop 4.18.10-arch1-1-ARCH #1 SMP PREEMPT Wed Sep 26 09:48:22 UTC 2018 x86_64 GNU/Linux

See also rust-lang/rust#54845

edit: This also leaks into JSON.parse, which can accept garbage strings even though ECMA-404 (the json standard prescribed for JSON.parse as defined in ECMAScript) only allows valid utf8 input.

[AljoschaMeyer]

Actually, straight up Buffer.from([255]).toString('utf8') already produces a string. Which is fine I guess, since js allows strings to contain garbage data, and the Buffer.toString does not specify failure modes (or their absence) at all. So this is not about wtf-8 at all.

This technically might not be a bug, just very surprising behavior that needs better documentation.

[mscdex]

FWIW even the TextDecoder web API defaults to the same behavior. Also other languages/platforms seem to follow suit as well (e.g. Java 8).

[SimonSapin]

Buffer.from([255]).toString('utf8').codePointAt(0).toString(16) returns fffd, which is the correct behavior: a U+FFFD RELACEMENT CHARACTER code point is used to represent part of the input byte sequence that was ill-formed in UTF-8. Same with 237, 166, 164.

[mscdex]

I would suggest using TextDecoder if you want to be able to catch (literally) decoding errors instead of having a string containing replacement characters.

Example:

new TextDecoder('utf-8', { fatal: true }).decode(Buffer.from([237,166,164]))
// throws exception

[AljoschaMeyer]

Thanks for the suggestion. I guess this mostly closes the issue, but it might make sense to specify that behavior (inserting U+FFFD in case of decoding errors) in the documentation of Buffer.toString.

And while I'm at it: the documentation explicitly links to the constant for the maximum string length, but does not specify what actually happens if the decoded data does not fit into a js string.

[BridgeAR]

I am marking this as good first issue to add a proper documentation for the described behavior.