npm causes the windows installer (.msi) to almost reach MAX_PATH (and using the .zip can easily exceed it)

[TheCloudlessSky]

• Version: current release (9.8.0) and LTS (8.10.0)
• Platform: Windows 7 64-bit

After installing Node 9.8.0 or 8.10.0 via the Windows Installer (.msi), the installation almost reaches the MAX_PATH allowed in Windows (260 characters). Here's the top 10 paths by length from the 8.10.0 installer:

$ cmd /c dir /S /B | Sort-Object { -$_.Length; } | Select-Object -First 10 | % { "$($_.Length) = $_" }
239 = C:\Program Files\nodejs\node_modules\npm\node_modules\libnpx\node_modules\yargs\node_modules\read-pkg-up\node_modules\read-pkg\node_modules\load-json-file\node_modules\parse-json\node_modules\error-ex\node_modules\is-arrayish\.istanbul.yml
239 = C:\Program Files\nodejs\node_modules\npm\node_modules\libnpx\node_modules\yargs\node_modules\read-pkg-up\node_modules\read-pkg\node_modules\load-json-file\node_modules\parse-json\node_modules\error-ex\node_modules\is-arrayish\.editorconfig
238 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-profile\node_modules\make-fetch-happen\node_modules\https-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\resolve.js
238 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-profile\node_modules\make-fetch-happen\node_modules\socks-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\resolve.js
238 = C:\Program Files\nodejs\node_modules\npm\node_modules\libnpx\node_modules\yargs\node_modules\read-pkg-up\node_modules\read-pkg\node_modules\load-json-file\node_modules\parse-json\node_modules\error-ex\node_modules\is-arrayish\package.json
237 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-profile\node_modules\make-fetch-happen\node_modules\socks-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\reject.js
237 = C:\Program Files\nodejs\node_modules\npm\node_modules\libnpx\node_modules\yargs\node_modules\read-pkg-up\node_modules\read-pkg\node_modules\load-json-file\node_modules\parse-json\node_modules\error-ex\node_modules\is-arrayish\.travis.yml
237 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-profile\node_modules\make-fetch-happen\node_modules\https-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\reject.js
237 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-profile\node_modules\make-fetch-happen\node_modules\http-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\resolve.js
236 = C:\Program Files\nodejs\node_modules\npm\node_modules\libnpx\node_modules\yargs\node_modules\read-pkg-up\node_modules\read-pkg\node_modules\load-json-file\node_modules\parse-json\node_modules\error-ex\node_modules\is-arrayish\.npmignore

I also use the .zip which can easily run into this problem if it isn't unzipped to a short enough path.

The .msi installer is very close to failing to install to C:\Program Files\nodejs if any module that npm depends on grows to a longer path. This isn't a problem when using npm since it'll properly dedupe the packages with npm > 3. This problem is with the installation of npm itself (which Node includes with its releases). Ideally, node_modules/npm would be pre-deduped so that it's not so long.

I plan to report this issue to the NPM team as well. For now, I've switched to using the standalone .exe and the yarn package manager (since it's just a single file and not a huge directory structure).

[bzoz]

Same issue was reported in #17409. Upstream issue was opened here npm/npm#19317, but there was no action taken. I think we should do something about this, I'm +1 on dedup, but I'm not an npm expert.

/cc @zkat

[TheCloudlessSky]

@bzoz I've tried running npm dedupe on the C:\Program Files\nodejs\node_modules\npm directory, and it didn't seem to help. I'm sure it's more complicated than this, however.

PS C:\Program Files\nodejs\node_modules\npm
$ cmd /c dir /S /B | Sort-Object { -$_.Length; } | Select-Object -First 10 | % { "$($_.Length) = $_" }
233 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\https-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\resolve.js
233 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\socks-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\resolve.js
232 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\http-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\resolve.js
232 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\https-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\reject.js
232 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\socks-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\reject.js
231 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\http-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\reject.js
230 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\https-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\race.js
230 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\socks-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\race.js
229 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\socks-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\all.js
229 = C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\http-proxy-agent\node_modules\agent-base\node_modules\es6-promisify\node_modules\es6-promise\lib\es6-promise\promise\race.js

[iarna]

For npm to flatten its dependencies, Node.js will need to fix a long standing bug in its installer where the new version of Node + npm is just extracted over the existing version. If we flattened deps in the new version, the deeply nested versions would still be on disk from the previous install and Node's module resolution algorithm would resolving those instead, causing confusing errors.

We first realized this was going on with the first release of npm@3, at which time we had to stop flattening.

[iarna]

If folks with this issue could try updating to 10.3.4, I think you'll find that it is resolved now (the installer related limitations on flattening were, in turns out, fixed last ~February, so the most recent version of npm is now maximally flat).

[TheCloudlessSky]

@iarna I couldn't find a release for 10.3.4 but I did install 10.4.0 (the latest) and it is much better 😄. I think we can close this.

PS C:\Program Files\nodejs> cmd /c dir /S /B | Sort-Object { -$_.Length; } | Select-Object -First 10 | % { "$($_.Length) = $_" }
133 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-client\node_modules\retry\test\integration\test-retry-operation.js
131 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-fetch\node_modules\cacache\node_modules\mississippi\package.json
131 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-fetch\node_modules\cacache\node_modules\mississippi\changelog.md
130 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-fetch\node_modules\socks-proxy-agent\test\ssl-cert-snakeoil.pem
130 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-fetch\node_modules\socks-proxy-agent\test\ssl-cert-snakeoil.key
129 = C:\Program Files\nodejs\node_modules\npm\node_modules\sorted-union-stream\node_modules\readable-stream\lib\_stream_passthrough.js
128 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-client\node_modules\retry\test\integration\test-retry-wrap.js
128 = C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-fetch\node_modules\cacache\node_modules\mississippi\readme.md
127 = C:\Program Files\nodejs\node_modules\npm\node_modules\sorted-union-stream\node_modules\readable-stream\lib\_stream_transform.js
127 = C:\Program Files\nodejs\node_modules\npm\node_modules\promise-retry\node_modules\retry\test\integration\test-retry-operation.js