propolis-package: produces output that omicron finds cromulent, despite missing propolis-server binary

[faithanalog]

I have some scripts I use which build propolis/crucible, package them as appropriate, and then generate a tuf repo with the standard tuf releng tool.

I encountered a benign compilation error with propolis, due to an out of date toolchain on my system.

error[E0658]: the `#[expect]` attribute is an experimental feature
   --> /home/vi/.cache/tuf-builder/crucible/upstairs/src/io_limits.rs:126:5
    |
126 |     #[expect(unused)]
    |     ^^^^^^^^^^^^^^^^^
    |
    = note: see issue #54503 <https://github.com/rust-lang/rust/issues/54503> for more information

Consequently, the propolis-server binary was never generated.

propolis-package did complain about this:

     Running `target/debug/propolis-package`
Error: Cannot append binary to tarfile: No such file or directory (os error 2) when getting metadata for target/release/propolis-server

However, the package output file was left in place:

vi@materia ~/.c/t/propolis (master)> tar tf out/propolis-server.tar.gz 
oxide.json
root/
root/var
root/var/svc
root/var/svc/manifest
root/var/svc/manifest/site
root/var/svc/manifest/site/propolis-server
root/var/svc/manifest/site/propolis-server/config.toml
root/var/svc/manifest/site/propolis-server/manifest.xml
root/
root/opt
root/opt/oxide
root/opt/oxide/lib
root/opt/oxide/lib/svc
root/opt/oxide/lib/svc/manifest
root/opt/oxide/lib/svc/manifest/propolis
root/opt/oxide/lib/svc/manifest/propolis/propolis.sh
root/
root/opt
root/opt/oxide
root/opt/oxide/propolis-server
root/opt/oxide/propolis-server/bin

My script proceeded to execute the tuf repo releng task, which accepted this as input. This was well up my scrollback of logs, and ultimately, I did not learn of the error until I tried to launch a VM, where found it in the "starting" state forever.

Now, truthfully, my script should have checked whether building the binary was successful and bailed before even running propolis-package. That said, in the event of errors like this, propolis-package ought to avoid placing an acceptable package at the location the tuf builder looks, perhaps by assembling the tarfile in a file with a different name, and renaming it only after it's done successfully.

[gjcolombo]

I think the partial-archive problem is a general issue with the omicron-package crate (it constructs output archives in place, the type it uses to construct archives finalizes them on drop, and there's no fallback logic to delete a partially-constructed archive). I filed oxidecomputer/omicron-package#68 for that behavior, since I think this could similarly befall any consumer of omicron-package.

I tried to launch a VM, where found it in the "starting" state forever.

@faithanalog Do you recall what Omicron commit your repo included? I'm not absolutely 100% surprised by this behavior, but I am a little surprised by it, since I would have guessed that omicron#6915 (merged October 30) would have allowed this instance to go to Failed (start saga asks to run the VM; this times out because the VM-start procedure blocks waiting for the Propolis server API to become available; start saga unwinds; before omicron#6915, the resulting request to terminate the Propolis zone would have blocked behind the task waiting for the API; after omicron#6915, the terminate request should jump the queue and the saga should unwind successfully). We might just want to try to reproduce this on Omicron main to get some fresh logs when we're ready to investigate further. (cc @hawkw FYI.)