HSL subprocess design considerations/approach

[fredemmott]

This document has suggestions for how to approach designing HSL subprocess, rather than a concrete suggestion for the design itself.

Requirements

• Secure by default
  • Does not use implicitly use the shell
  • Does not require users to escape arguments; e.g. require vec arguments, pass directly to builtins as a vector.
  • Does not internally escape arguments in a way that depends on undefined behavior, e.g. shell escaping rules.
  • Safe default values
• Non-blocking, i.e. use awaitables to wait for process exit
• Feels consistent with the rest of the HSL
  • Naming
  • Prefer free functions over classes where possible
  • Interacts cleanly with HSL IO (e.g. for process stdin/stdout)
• Even the low-level public API should feel reasonably usable
  • Good default values (but safety takes priority over usability if a compromise is needed)
  • Should be designed to be as friendly as practical for direct use, not just as a building block for the higher-level API
  • Easy to use safely
  • Possible - but hard - to use 'incorrectly'
• Support completely arbitrary FD combinations, including spawning with no FDs at all
• Usable as a 100% replacement for PHP builtins, including shell_exec, proc_open, etc
  • NOT drop-in or wrapper-based replacement - can require callers are refactored
    does not require interoperability with PHP resources; e.g. removing proc_open from www would require changing existing callsites to replace their usage of PHP resources for pipes with HSL IO.
    Does not require removal of existing PHP builtins, just the potential to migrate to them
  • “100% replacement”: there should be a plausible (even if difficult) path to replacing all existing use cases of the PHP subprocess builtins with the new HSL functions. No “sorry, the HSL doesn’t support that, you need to use the PHP builtins instead”
• High-level wrapper APIs
  • base on existing uses, e.g. shell_exec, csprintf() and execx()
  • keep safety at at least the current levels in FB www
  • These convenience APIs should still feel consistent with the rest of the HSL

Compare with other languages

I strongly recommend looking at other modern languages/frameworks to see how they address subprocesses; I would start by looking at Rust, Scala, Python3, Ruby, folly::subprocess (C++) and QProcess from Qt (also C++). A full comparison with each isn’t needed, and you don’t have to base the design on one of them, but they’re useful to learn from. Look for:

• Overall API design
• Pitfalls their design avoids, or mentions in their documentation
• Have they done very large breaking API changes/redesigns (e.g. for HSL OS, OS exception hierarchy in Python2 -> Python3)? If so:
  • Why?
  • What was the community feedback for this change?
  • Separate feedback into ‘change is bad/the improvement wasn’t worth the pain’ (not so useful for us when designing a new API rather than modifying an existing one), ‘the new thing is better’, and ‘the old thing was better’

For example, HSL IO can be considered a more strictly-typed hybrid of Rust and Java’s IO APIs, but with Python3’s IO/OS exception model.

References

Async wait for exit: facebook/hhvm#8942

Handling of STDIN: FB-only link. In short, many programs do not handle FDs 1/2/3 being closed well, especially when an open()/connect()... allocates FD1 for something else. Safer to default to having the FDs open, but attached to /dev/null

Other people to talk to

• (various FB people)
• #HSL in hacklang.slack.com