bug: no such file or directory

[Arqu]

Since the merge of the persistence PR I've been consistently getting Error: No such file or directory (os error 2) and just terminating the app locally and on the CI (netsim) runners.

Error/Trace

2023-03-21T12:04:19.168111Z  INFO iroh::provider::database: Loading snapshot from /Users/arqu/Library/Application Support/iroh...
Error: No such file or directory (os error 2)

Stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: anyhow::backtrace::capture::Backtrace::create
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.69/src/backtrace.rs:216:13
   3: anyhow::backtrace::capture::Backtrace::capture
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.69/src/backtrace.rs:204:17
   4: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.69/src/error.rs:547:25
   5: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/result.rs:2105:27
   6: iroh::provider::database::Snapshot<std::io::error::Error>::load
             at ./src/provider/database.rs:98:21
   7: iroh::provider::database::Database::load_internal
             at ./src/provider/database.rs:208:24
   8: iroh::provider::database::Database::load::{{closure}}::{{closure}}
             at ./src/provider/database.rs:225:49
   9: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/task.rs:42:21
  10: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
  11: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  12: tokio::runtime::task::core::Core<T,S>::poll
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  13: tokio::runtime::task::harness::poll_future::{{closure}}
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  14: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panic/unwind_safe.rs:271:9
  15: std::panicking::try::do_call
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492:40
  16: ___rust_try
  17: std::panicking::try
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:456:19
  18: std::panic::catch_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panic.rs:137:14
  19: tokio::runtime::task::harness::poll_future
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  20: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  21: tokio::runtime::task::harness::Harness<T,S>::poll
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  22: tokio::runtime::task::raw::poll
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:255:5
  23: tokio::runtime::task::raw::RawTask::poll
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  24: tokio::runtime::task::UnownedTask<S>::run
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:431:9
  25: tokio::runtime::blocking::pool::Task::run
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:159:9
  26: tokio::runtime::blocking::pool::Inner::run
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:511:17
  27: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /Users/arqu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:469:13
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/backtrace.rs:122:18
  29: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/mod.rs:514:17
  30: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panic/unwind_safe.rs:271:9
  31: std::panicking::try::do_call
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492:40
  32: ___rust_try
  33: std::panicking::try
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:456:19
  34: std::panic::catch_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panic.rs:137:14
  35: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/mod.rs:513:30
  36: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:248:5
  37: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/alloc/src/boxed.rs:1940:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/alloc/src/boxed.rs:1940:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys/unix/thread.rs:108:17
  38: __pthread_deallocate

[rklaehn]

So the issue itself can be easily fixed by not failing if the directory exists but the db file is not in there. Then you just create it.

The question is: should we do something more clever when finding an old iroh dir, like refusing to start?

[Arqu]

Tbh we're so early in establishing "our dir" that going the simplest, least friction route is probably best as we will probably change stuff around in the near term.

[Arqu]

After some manual scrubbing, seems like this will be a hard requirement for netsim.

[dignifiedquire]

@Arqu @rklaehn can you two get this figured out and fixed please?

[Arqu]

Yeah, don't have a real clue on why it keeps happening. I used to encounter it a lot on CI. Nowadays I just delete the persisted folder and move on. It seems to come from unexpected starts/stops of iroh and it's persistence procedures. It's been like this since the first PR for it.

@rklaehn lmk how/if I can add anything that helps debugging this.

[rklaehn]

@Arqu is this still happening with the new new persistence?

[Arqu]

Afaik, it's good now been a while since we dropped that nuke dir line from netsim. I still have to get to implementing proper use of IROH_DATA_DIR, but other than that haven't seen any issues recently.

[ramfox]

@Arqu, think we can close this now?

[Arqu]

Yes! Done in n0-computer/chuck#30