Flexible stdios for IPC

[gireeshpunathil]

• Version: master
• Platform: all
• Subsystem: child_process

A noted problem pattern is broken data flow when i) spawning OS commands that predate Node.js and that do not expect their stdios to be non-blocking. ii) Composite IPC that involves more than one Node process in the process chain.

The underlying issue is the mixed mode I/O in a number of variations based on number of parent-child combo that is possible.

While many of these can be mitigated by studying the I/O characteristics of parent and child and crafting a suitable IPC that works, inability to (programatically) control the blocking behavior (flow mode) of the stdio channel causes usability issues.

Proposals:

Node parent -> non-Node child
Node parent - Node child

• exec family functions to receive option to control the blocking behavior of child stdios (new option)
• adjust the child stdios accordingly, while retaining parent side of pipes non-blocking
• convert stdout and stderr to nonblocking prior to exit to help drain the buffered chunks (revisit process: flush stdout/stderr upon process.exit() #6773)

Node parent - non-Node child -> Node grand child
Node parent - non-Node child -> non-Node grand child

• restore stdio characteristics on exit (already being worked upon in src: restore stdio on program exit #17737)
• convert stdout and stderr to nonblocking prior to exit to help drain the buffered chunks

references:
truncated stdout on porcess exit: #19218 #784 #6456 #6379
pre-fabricated duplex streams for IPC: #16553 #15714 #13542
inconsistent IPC with process chain: #9413
unexpected flow mode for stdio: #14752 #18446 #19838

[bnoordhuis]

exec family functions to receive option to control the blocking behavior of child stdios (new option)

No. If fds in child processes are non-blocking, then that's simply a bug.

[gireeshpunathil]

@bnoordhuis - I suggest we:

• cover the common combinations of the process chain based on usage
• for each set, define what is expected and what is a bug (this outlines a specification, that can freeze with ratification from the eco system channels, if need be)
• implement this sepc - align existing capabilities or fix bugs
• document the behavior.

It may be that we have one or two minor issues that need fixing, but given the number of open (and closed) issues on this topic, I guess having a child_process specification document helps users at large.

[gireeshpunathil]

unexpected flow mode for stdio: #14752 #18446 #19838 is fully covered.

[jasnell]

Ping @gireeshpunathil ... does this need to remain open? Is there more to do?

[gireeshpunathil]

With many stable versions released and used in the field with the current behavior, and we haven't heard any issues related to these topics in the recent past, I would assume that the applications have adapted to the way Node.js handles stdio. We can always revisit this if it is reported as problematic.

If we all are in agreement, then we could not only close this, but all the referenced issues too. I would like to seek consensus: @nodejs/child_process