So can we just expect POSIX these days?

[jpco]

There are a few places in es' codebase where it currently fusses about working right in this-or-that buggy or incompatible UNIX from back when that was the way the world looked in the early '90s. These days it seems to me that among the various extant UNIXoid OSes, POSIX support has gotten pretty good and coding es to POSIX APIs would be pretty reasonable.

A couple specific examples I have in mind:

• Can we just expect to be able to use waitpid(3) in proc.c? It would be nice to be able to unify on a single system version of wait.
• Can we just expect to be able to use setpgid(3) in prim-sys.c? The current behavior with setsid(3) is actually (IMO) a bug given how differently the two functions behave under POSIX.

I'm not trying to yank out current compatible behavior just for the heck of it, but it would be nice to not have to do a, say, HAVE_WAITPID test for what seems like it should be extremely standard functionality.

Does anyone have a strong reaction to this? Is anyone using es on a not-very-POSIX machine?

[wryun]

I'm a bit torn about this.

I don't think we need to care too much about non-POSIX support - if anyone turns up who desperately has to run es on some bizarre OS, they can easily grab an older version - but there's a bit of 'if it isn't broke, don't fix it'. Any time you touch a mostly untested code-base there's the chance that you break something in some strange/unexpected way... but I'm not the one trying to actually work with the code, and haven't been looking at the PRs, so if you'd enjoy doing do some careful cleaning up I'm not going to complain.

[jpco]

I don't think I mean to do any clean-up just for the sake of clean-up, but -- to look more into my motivating examples here...

Autoconf, which is basically our best authority on portability right now, seems upset about using wait3() because it's been deprecated since 1997 (!), completely nonstandard now, and liable to be removed any time. So we should probably get ahead of that and just replace it (or at least stop using it by default). waitpid() is the usual choice of replacement for shells because it provides a lot of the nice fiddly wait options that a shell likes to use, and it's specified in POSIX. But AFAICT waitpid() is (on es time scales) relatively new, and for maximum portability, some tools will define a fake waitpid()-alike that uses wait() on the backend, in case waitpid() isn't available.

But a fake waitpid is some 100 lines of extra code to maintain, plus autoconfed-feature-flag logic to select, and, worst, it's a kludge, since waitpid() can do strictly more than wait() can. So it would reduce complexity quite a bit to just say "we're doing waitpid()! It's standard."

The setpgid() case is somewhat similar. Right now es defines it in a macro to mean setsid() for any systems which have setsid() (which, given setsid() is in POSIX, should be most of them). But this causes a couple of unexpected behaviors for people who want to run background commands on POSIX OSes -- for example: unlike just about every other shell, you can't actually see a backgrounded command with a simple ps! That's weird enough for me to consider it a bug, especially since it doesn't contribute to the simplicity or extensibility of the shell in any way.

Trying to fix this in a way that works for all the UNIXen all the time is complicated, though: back in the day, SysV and BSD created their own incompatible ways to group processes, and the POSIX spec tried to come up with something new that kinda-sorta smooshes the two together, but which more closely matches the BSD way. So the most-correct way to do things seems to be: "do it the POSIX way, unless you can't, in which case do it the BSD way, unless you can't, in which case do it the SysV way." But this is complicated by itself, I don't know how to correctly test for it (autoconf docs aren't much help since they mostly say "everything is POSIX"), and it has subtle semantic implications that I don't understand (what happens on every one of these systems when a backgrounded job reads from the terminal? Outside of POSIX, I couldn't tell you.)

So in both of these cases we know how to strive for absolute, maximum portability, but it's strictly harder to implement and I can't even guarantee non-POSIX systems will get the only-mostly-correct behavior I expect, because I can't really find any!

[memreflect]

I can't even guarantee non-POSIX systems will get the only-mostly-correct behavior I expect, because I can't really find any!

MINIX 3 is doing its own thing, kinda POSIX and kinda not, yet it's still Unix-like.
For example, it has setsid() and waitpid(), but it's missing setpgid() and tcsetpgrp() (it can use the TIOCSPGRP ioctl in place of tcsetpgrp() as es already does) — all of those interfaces have all been standard since POSIX.1-1990 or earlier.
That said, i don't know anybody who actually uses MINIX, so i'm not sure it's worth the trouble.

Plan 9/Inferno OS is even more independent, not even having signal() or sigaction(), so i wouldn't bother wasting time with getting es to work on it.

POSIX_ME_HARDER=1

[jpco]

I think that it could be an interesting experiment to try to separate out the "core" es stuff (gc, syntax, eval, variables, primitives, exceptions) from the POSIX-oriented es stuff (fork/exec/wait, signals, tty, environ). Then once the POSIX stuff is factored out, one could define alternatives to it to "nativize" es to operating environments like Plan 9 (rfork, notes, no tty, /env) or Haiku (teams, messaging, etc. -- you know, I've seen the Haiku folks recently wishing they had a better application-scripting language...) I've also wondered, and this would be a ridiculous nightmare to actually implement, how it could look to swap out the traditional fork/exec pattern in es for a posix_spawn/pthread model. Hypothetically, at the user level, it could just require re-defining the same hooks (%run, %fork, etc.) in terms of each of the "OS-native" primitives to maintain consistency.

Come to think of it, this is basically the "embedded es" concept that Paul and Byron were starting to work on, just from the point of view of OS semantics rather than Tcl-style embedding. I just think it could be cool to see how gracefully es could be adapted to these environments.

[memreflect]

I tend to lean heavily in favor of careful removal of legacy code instead of leaving it to bitrot.
Even Solaris 10 will be EoL in 2027 after a whopping 22 years, so i don't see any reason why es should support platforms that are far older like DEC Ultrix.

Does es even work on some of these platforms still?
I managed to get it to build on an OPENSTEP 4.2 VM as a test, but that was only after configure choked on grep -E (it's spelled egrep there), and i had to shorten the list of signal files in config.status to fit the line length limit so it could spit out a proper Makefile.
Unlike NeXTSTEP, OPENSTEP also lacks a POSIX library, so some stuff like sigaction() in the man pages just wasn't available.
While it's nice that it still builds with relatively little difficulty, OPENSTEP is ~30-year-old abandonware at this point.
If even Solaris 10 is EoL after little more than 20 years, why should es support any OS that is even older?

[jpco]

Relatedly, I just discovered that es seems to mostly work fine on Haiku (impressive!), except that Haiku encodes WIFEXITED/WIFSIGNALED/WIFSTOPPED differently than Linux and friends, meaning that, for example, any normal falsey exit status gets interpreted as a WIFSIGNALED:

; echo <={es -c 'result 1'}
hangup
sighup
; echo <={es -c 'result 2'}
sigint

and so on. I don't believe POSIX actually specifies how exit statuses should be encoded in wait() -- might be something to be fixed here.

[memreflect]

For portability's sake, i would say fix the SIFSIGNALED macro and friends or just drop them in favor of the standard WIFSIGNALED macro and such.

One important difference concerns core dumps — GNU/Linux, the BSDs, Solaris, and HP-UX all use WCOREDUMP(status), and this was added to XPG8/POSIX.1-2024, but Haiku uses its own WIFCORED(status).
Before the recently published POSIX.1-2024, there was no portable way to determine whether a process created a core image or not.
Several systems other than Haiku seem to prefer setting the 0x80 bit as the core dump flag.
Is it reasonable to retain the current assumption that (status) & 0x80 indicates core was dumped as a fallback, or should this information simply be omitted from the signal info in the event that no macro exists to reliably obtain this information for the sake of avoiding issues like you ran into on Haiku?

[pmarreck]

I would not remove "effectively dead" code unless there is good test coverage, because it's effectively a refactoring. As @wryun mentioned, there is unknown risk associated with all code modifications, even the ones that look reasonable. Doubly so without great test coverage.

I've harped on proper test coverage for es before (was it... a few years ago now?), and did some work on my own fork (which I believe I got a bit stuck on the finer details of merging in all the neat feature branches it's had, not being a C coder by day... side note, I wonder if codeium windsurf or cursor IDE with LLM assistance might help?), so maybe I should abandon that for now and just help with proper test coverage for the main ES repo (i.e. this one).

Assuming I can find the time. 3.5 year old toddlers steal a lot of it...

This is such a neat and underrated shell IMHO, though. My latest shell-envy has been of newer shells like NuShell and Murex that permit piping of structured data instead of unstructured strings, but (like most everything unfortunately) they're not particularly "functional"

Anyway, I think expecting POSIX is fair. At least POSIX is a standard...

[jpco]

I've harped on proper test coverage for es before (was it... a few years ago now?), and did some work on my own fork

If you have commits containing better test coverage that you'd be willing to share, I'd be happy to try upstreaming them. I should really get to work on better test coverage myself and I'm feeling a bit guilty about that. (I've even been merging more changes without tests included! Terrible!) At least we've been working on getting stricter compilation and more standards compliance working, that's an improvement at least. Actually, once #163 is merged and the stricter flags are integrated into testing/building, I think the "POSIX-ification project" is basically done! At which point I promise I'll get to writing more tests......

My latest shell-envy has been of newer shells like NuShell and Murex that permit piping of structured data instead of unstructured strings

I do think this would be an interesting direction to explore for es. I'm not picturing this being facilitated through pipes in es (personally, I prefer pipes stay pipes and structured data use another mechanism) but some extension of es' native exceptions/rich returns/arguments seems like it should work.

I suggested something with this in mind in #67 but I think if we wanted to make it real and worthwhile we'd want to consider first-class environments (which aren't such a crazy idea -- they're something the original authors were considering) as well as something like python's yield, which could also be related to how %read works, plus some potential syntax support. But now I'm daydreaming, and I don't have a broad enough base of experience with different functional languages to have a clear idea of what would work best for es.