Propolis server crashed due to crucible panic after downstairs client checked_state_transition

[askfongjojo]

An instance on rack2 (deep41) failed to come up after the rack was updated to the omicron commit db656d750a888724b672776fe9b2351c4a683d64. It left behind a bunch of propolis core files (there are many because the auto-restart background task has been retrying the boot). I inspected one of the core files:

BRM44220005 # mdb /pool/ext/4eb2e4eb-41d8-496c-9a5a-687d7e004aa4/crypt/debug/core.oxz_propolis-server_3f560951-4bf8-4e23-b1f1-46b47ea71557.propolis-server.10039.1736876977
Loading modules: [ libumem.so.1 libnvpair.so.1 libc.so.1 ld.so.1 ]
> ::status
debugging core file of propolis-server (64-bit) from oxz_propolis-server_3f560951-4bf8-4e23-b1f1-46b47ea71557
initial argv: /opt/oxide/propolis-server/bin/propolis-server run /var/svc/manifest/site/propo
threading model: native threads
status: process terminated by SIGABRT (Abort), pid=10039 uid=0 code=-1
> $C ! demangle
fffff5ffec3fec70 libc.so.1`_lwp_kill+0xa()
fffff5ffec3feca0 libc.so.1`raise+0x22(6)
fffff5ffec3fecf0 libc.so.1`abort+0x58()
fffff5ffec3fed00 0x28c9ab9()
fffff5ffec3fed10 0x28c9aa9()
fffff5ffec3fed70 rust_panic+0xd()
fffff5ffec3fee30 std::panicking::rust_panic_with_hook::h503ea5292ea6f2f4+0x231()
fffff5ffec3fee70 std::panicking::begin_panic_handler::{{closure}}::h2eb8efd06bcdc46a+0x98()
fffff5ffec3fee80 0x28b0699()
fffff5ffec3feeb0 0x28b2d3c()
fffff5ffec3feee0 0x28f7f6f()
fffff5ffec3fef60 0x179caaf()
fffff5ffec3ff050 crucible::client::DownstairsClient::checked_state_transition::h79eba992c72a0ae7+0x5b1()
fffff5ffec3ff0d0 crucible::client::DownstairsClient::reinitialize::h10fbf646c0b356a9+0x82()
fffff5ffec3ff100 crucible::downstairs::Downstairs::reinitialize::h0e4cfa17ec60f3d7+0x6c()
fffff5ffec3ff610 crucible::upstairs::Upstairs::apply::ha37352906560c12c+0xca1()
fffff5ffec3ffa10 crucible::up_main::_$u7b$$u7b$closure$u7d$$u7d$::h8beba88cfcd01f69 (.llvm.4294170658799645515)+0x900()
fffff5ffec3ffa70 tokio::runtime::task::core::Core<T,S>::poll::h7152226d1b2d68ad+0x3a()
fffff5ffec3ffad0 tokio::runtime::task::harness::Harness<T,S>::poll::h86bcf5a3f3cecd0f+0x5b()
fffff5ffec3ffb10 tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h0ecaaf1c37889b4a+0x100()
fffff5ffec3ffbb0 tokio::runtime::scheduler::multi_thread::worker::Context::run::h322af521d260d153+0x5bb()
fffff5ffec3ffc10 tokio::runtime::context::scoped::Scoped<T>::set::h4fb1c8f547a0b41a+0x2a()
fffff5ffec3ffcc0 tokio::runtime::context::runtime::enter_runtime::h60c5d270d5568811+0x19a()
fffff5ffec3ffd00 tokio::runtime::scheduler::multi_thread::worker::run::haafe064344917608+0x7c()
fffff5ffec3ffd70 tokio::runtime::task::core::Core<T,S>::poll::h172cb272e4c8592a+0x72()
fffff5ffec3ffe00 tokio::runtime::task::harness::Harness<T,S>::poll::h9bcb9426c88706d2+0x8a()
fffff5ffec3ffeb0 tokio::runtime::blocking::pool::Inner::run::he438bb77f2152818+0xe4()
fffff5ffec3ffef0 std::sys::backtrace::__rust_begin_short_backtrace::h19e968f4e0e0c184+0x3e()
fffff5ffec3fff60 core::ops::function::FnOnce::call_once{{vtable.shim}}::h07400e25b9ff8886+0x93()
fffff5ffec3fffb0 std::sys::pal::unix::thread::Thread::new::thread_start::he13a45effb26dfc6+0x2b()
fffff5ffec3fffe0 libc.so.1`_thrp_setup+0x77(fffff5ffeef23240)
fffff5ffec3ffff0 libc.so.1`_lwp_start()

[benjaminleonard]

Core files have been moved to dogfood-mupdate-20253001 in the staff folder after todays dogfood update.