A new design for the old monster: job control

[jpco]

if someone comes to me and Byron and says, ``i need these two extra primitives and this one extra line of conditionally compiled code in signal.c and i'll then be able to write full csh-style job control in es,'' we'd probably put it in, albeit with some serious grumbling. (i don't think it will happen, though. it just don't look that easy to me.)

• Paul Haahr (http://wryun.github.io/es-shell/mail-archive/msg00133.html)

One of the problems with the existing job control branch (aside from being extremely out of date) is that it isn't very... extensible. I would expect a job control that can be enabled by running . job-control.es in .esrc.

I've been playing around with this for a bit and I've sketched up a new design for supporting job control. Here are the major new behaviors that I think I need to make it work:

• New syntax +signame for the signals variable, which causes all processes in the shell's process group to ignore the signal, but sets it to sig_default in child process groups. (This is like /signame but at a process-group level).

• Extended behavior for $&newpgrp: if it is called with arguments, then it runs those arguments as a command, but it puts any new processes forked by that command into a single new process group. This would be how "jobs" are created, because "jobs" are essentially just process groups and a little shell sugar.

• Use the %run hook function for running binaries. #90, to allow fn-%run = $&newpgrp $&run. Other ways to fork/exec a binary are already hookable -- for example, fn-%pipe = $&newpgrp $&pipe.

• Extension to $&wait so that it can take two new flags changing its behavior: -n would correspond with the WNOHANG option in waitpid(3) and -c would cause $&wait to send a SIGCONT to the passed pid (or -pgid) before waiting. (For the -n flag, Add a few things to make $&waiting more flexible. #149)

• Add a few things to make $&waiting more flexible. #149 - A new hook function %report-status, which does the job that printstatus() does today -- essentially, it takes information about a waited-for child process (or process group) and decides whether to print a message about "quit--core dumped" or the like. A hook function makes this more flexible to be able to account for stopped processes and process groups, and allows the shell to do things like save the last stopped process groups to make commands like fg (with no arguments) possible.

• (Optional, but nice) Extended behavior for $&apids: if given a pid or -pgid as an argument, it returns all alive child processes which correspond with that argument. This allows for an easy way to map from -pgids to pids for external binaries which may not understand process groups.

• (Done automatically by the shell when -DJOB_PROTECT=1, with Use POSIX-compatible setpgid() #142) New primitive(s) which allow es to reset its process group (and more importantly the terminal's process group) to its original value before exiting. Not having this can make it so that on exit, the first moment some input is sent to the parent process, the whole terminal will exit, because the parent process's process group is no longer in the "foreground" according to the terminal.

Follow-on/internal changes also required:

• (Done with Posix and portability improvements #128) Move from wait3() to waitpid()+getrusage().

• Extend ewait() to wait for all processes in a process group, to foreground a process group if appropriate, and to return all the exit statuses from the process group in one list. (This generalizes what $&pipe does today.) Also teach ewait() that stopped child processes exist.

• Remove the special-case logic in $&pipe to wait for every process individually, if the child processes are being managed in a process group (in which case ewait() will wait for the whole group automatically).

• Remove the special-case newpgrp() logic in $&background, if the child processes are already being newpgrp()ed by $&newpgrp. Also remove the macro that defines setpgrp as setsid, because those are two entirely different things (can anybody explain to me if there was ever an OS where that redefinition makes any sense?) (The second part done with Use POSIX-compatible setpgid() #142).

• Extend $&apids to correctly account for child process groups.

This ends up adding up to a fair number of lines of code, especially in ewait() which needs a lot of new explicit and implicit behaviors. However, it's not very much additional conceptual load for the user, I think it can be done in a backwards-compatible way such that it is almost no conceptual load for the non-job-control user, and it should allow the . job-control.es invocation I imagine.

A couple sort of meta changes which would help make the . job-control.es aspect easier (as opposed to just hard-coding everything in initial.es) are:

• (Done) Add canonical extensions #131, which enables es to be built with pre-defined scripts like job-control.es (since a less-than-expert user is not going to simply come up with that script without some degree of assistance).

• Additional support for hacking up %interactive-loop. In particular, ways to mess with signal handling and other exceptions without completely redefining the %interactive-loop function, and a clearer pattern for "fn-X should be defined as A in the interactive loop, but B in the batch loop", preferably without using if {%is-interactive}.

See https://gist.github.com/jpco/cd5caccc344e2e59a254b8bcae9e7e23 for a simple demonstration job-control.es. In this version, grouped processes are represented with the negative number -pgid, which follows the convention already used in waitpid() and kill(). Users who want more csh-style job control, where each job gets its own shell-internal number and such, can extend %report-status to do that assignment.

(Some of the changes may be useful outside the context of job control -- I'd love to get rid of all the whining about wait3(). Also, @memreflect, I think it's this %report-status hook function that would let you flip the sense of which exit statuses are true and which are false :)

[jpco]

So, clearly, I've already been working towards this. :) My ideal is that each component of this design is useful enough to add even without being justified by the larger "job control" idea, since I know that for some people, job control isn't actually useful. So far, I think the PRs that have already been merged meet this bar. I think that the %run hook pretty clearly does as well -- in fact, I wrote #80 before even coming up with this idea for job control.

My speculative long-term plan is basically this, if I can get people to approve of each step along the way:

• Get Use the %run hook function for running binaries. #90 and Add a few things to make $&waiting more flexible. #149 merged.
• Create a PR for making %interactive-loop more flexible.
• After the two outstanding PRs are merged, create a large-ish PR which basically teaches es to handle process groups, backgrounding, foregrounding, stopped children, and all that. Unfortunately, right now, I don't see a great way to break that PR down in a way that would make it feel "complete". You kind of just need the process-group stuff to happen all at once.
• After those two not-created-yet PRs are merged, create a PR to add a job-control.es canonical extension script. At that point I'd consider "job control in es" complete!

However, like I said before, I want each of these PRs to be useful in their own right, and that means that even if at some point in the middle of this to-do list people say "okay, cut it out", it shouldn't be necessary to revert everything that's already been merged. So if this plan doesn't work out, it shouldn't leave stuff a mess.