Alpine 3.8 cluster failures

[rvagg]

I just added Alpine 3.8 to CI and removed 3.6 in the process, shifting alpine-last-latest-x64 to Alpine 3.7 and giving alpine-latest-x64 to Alpine 3.8. It tested well on my local machine across our major branches, but now it's enabled in CI we're getting consistent errors with on all test runs:

15:57:44 not ok 647 parallel/test-cluster-master-error
15:57:44   ---
15:57:44   duration_ms: 120.63
15:57:44   severity: fail
15:57:44   exitcode: -15
15:57:44   stack: |-
15:57:44     timeout
15:57:44   ...
15:57:44 not ok 648 parallel/test-cluster-master-kill
15:57:44   ---
15:57:44   duration_ms: 120.92
15:57:44   severity: fail
15:57:44   exitcode: -15
15:57:44   stack: |-
15:57:44     timeout

What should we do with this? Do I remove it from CI for now or can someone spend some time investigating?

The Dockerfile is here minus the template strings which you can easily stub out to set this up locally if you want to give that a go. I'm not sure what the difference is with my local machine but perhaps I wasn't testing it on the latest master and there's something new or perhaps there's a Docker differential.

[Trott]

Quick thing to try: Does moving the tests to sequential cause them to pass in CI on this platform? (I would create a branch and try myself but I've got non-computer things to focus on for the next few hours.)

[Trott]

(Also: Our code now prints stdout and stderr on timeouts so someone can try adding a bunch of console.log() statements to see where things are hanging up.)

[refack]

Testing move to sequential
https://ci.nodejs.org/job/node-test-commit-linux/20716/

[refack]

BTW: before switch to 38 tests took < 2s

[refack]

Testing move to sequential
https://ci.nodejs.org/job/node-test-commit-linux/20716/

https://ci.nodejs.org/job/node-test-commit-linux/20716/nodes=alpine-latest-x64/console
Still fails.

[Trott]

Running with some logging: https://ci.nodejs.org/job/node-test-commit-linux/20725/nodes=alpine-latest-x64/console

[Trott]

The logging shows that the test succeeds except for that pollWorkers() in both tests never detects that the PID exits. I wonder if the bug is in common.isAlive(pid). That code uses process.kill(pid, 'SIGCONT'); so maybe there's something about SIGCONT and permissions or something in the Docker container?

[Trott]

@nodejs/docker

[Trott]

common.isAlive(pid) expects process.kill(pid, 'SIGCONT') to throw if pid does not exist. @rvagg @refack or someone who knows how to log into the Docker container running on CI, any chance you can see if ./node -e 'process.kill(12345)' throws or not (after checking that there is no pid 12345 running)?

[maclover7]

bash-4.4$ ./node -e 'process.kill(12345)'
internal/process/per_thread.js:194
      throw errnoException(err, 'kill');
      ^

Error: kill ESRCH
    at process.kill (internal/process/per_thread.js:194:13)
    at [eval]:1:9
    at Script.runInThisContext (vm.js:88:20)
    at Object.runInThisContext (vm.js:285:38)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at evalScript (internal/bootstrap/node.js:563:27)
    at startup (internal/bootstrap/node.js:248:9)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:596:3)

Also, @Trott (or anybody else with the nodejs_build_test SSH key), I had written up some docs about how to restart/SSH into the CI docker containers (in case more testing is needed, and you want it to be self-service :))