- Version: 10.12.0
- Platform: Windows 10 64-bit
- Subsystem: fs
PR #5616 gave us support for Buffer paths in all fs methods, primarily to allow interacting with files of unknown or invalid file encoding. This helps on UNIX/Linux where filenames are technically just strings of bytes and do not necessarily represent a valid UTF-8 string.
Similarly, on Windows, filenames are just arrays of wchars, and do not necessarily represent a valid UTF-16 string, however the current { encoding: 'buffer' } variety of fs methods do not properly handle this case. Instead, the Buffers that are returned are UTF-8 representations of (potentially losslessly / incorrectly) decoded UTF-16 filenames. Similarly, it's not possible to pass as input Buffers that represent the raw UTF-16 bytes. This leads to the possibility of files that Node can't interact with at all.
Consider the following code that makes a file that doesn't have a proper UTF-16 name. The created file can be seen and interacted with using Windows Explorer and Notepad without issue.
#include "stdafx.h"
#include <iostream>
#include <windows.h>
#include <string>
using namespace std;
int main()
{
// Junk surrogate pair
const wchar_t *filename = L"hi\xD801\x0037";
HANDLE hfile = CreateFileW(filename, GENERIC_READ, 0, NULL, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, NULL);
return 0;
}Then, running the following Node code in the same directory shows that the file cannot be accessed:
const fs = require('fs');
const bufs = fs.readdirSync('.\\', { encoding: 'buffer' });
for (const buf of bufs) {
try {
const stat = fs.statSync(buf);
console.log('successfully got stats of: ' + buf.toString('utf8'));
} catch (err) {
console.log('error getting stats of: ' + buf.toString('utf8'));
}
}The above code produces the following output when run in the same dir as the invalid UTF-16 file:
error getting stats of: hi�7
successfully got stats of: test.js
Refs:
#5616
rust-lang/rust#12056
jprichardson/node-fs-extra#612
PR #5616 gave us support for Buffer paths in all fs methods, primarily to allow interacting with files of unknown or invalid file encoding. This helps on UNIX/Linux where filenames are technically just strings of bytes and do not necessarily represent a valid UTF-8 string.
Similarly, on Windows, filenames are just arrays of wchars, and do not necessarily represent a valid UTF-16 string, however the current
{ encoding: 'buffer' }variety of fs methods do not properly handle this case. Instead, the Buffers that are returned are UTF-8 representations of (potentially losslessly / incorrectly) decoded UTF-16 filenames. Similarly, it's not possible to pass as input Buffers that represent the raw UTF-16 bytes. This leads to the possibility of files that Node can't interact with at all.Consider the following code that makes a file that doesn't have a proper UTF-16 name. The created file can be seen and interacted with using Windows Explorer and Notepad without issue.
Then, running the following Node code in the same directory shows that the file cannot be accessed:
The above code produces the following output when run in the same dir as the invalid UTF-16 file:
Refs:
#5616
rust-lang/rust#12056
jprichardson/node-fs-extra#612