Some backend (i.e. opendal:*, rclone) implementations are incompatible with async features (i.e. Webdav) which result in runtime crashes

[lostb1t]

When browsing to a specific snapshot the application crashes (using opendal S3)

The application panicked (crashed).
Message:  Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.
Location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/opendal-0.44.2/src/layers/blocking.rs:264

[nardoor]

To keep you updated, I could reproduce the issue using opendal:fs.
The fix is not trivial, I'll keep this issue updated when I reach it.

[enboig]

Same problem using sftp

[nardoor]

I renamed the issue to be more generic.

Problem insights:

To give some insights the problem is the following:

1. (rustic_core) All our backend traits (and implementations) in rustic-rs/rustic_core are sync
2. (rustic_core) Some crates used to implement these backends are async but provide a sync compatibility using block_on. This compatibility impl is used to impl the backend traits.
3. (rustic_rs) Webdav feature deploys an async server
4. (rustic_rs) When using Webdav with backends that use block_on (see 2.) we do the following ~callstack:

- main
- tokio runtime for webdav
- async webdav server handle
- opendal backend read
- `block_on` blocking operator read
- runtime block on <crash>

Remediation, fixing:

I see 2 main steps to advance on that issue:

1. (~ simple) avoid crash by disabling incompatible configuration:

• list async feature incompatible backends and fail early when using webdav with them
• make sure rustic_rs as no other async features such as webdav

2. see with the team for the bigger fix:

• refactor backend traits into async traits? this is huge breaking change and a lot of work.. (in a nutshell, all rustic_core and rustic_backend would need a ~rewrite.
  howevre to me it would make sense because the backend operations are async by nature (network, disk, ...)
• some other way? ...

[nardoor]

Using rclone as backend with webdav feature result in a similar (but different) crash:

The application panicked (crashed).
Message:  Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
Location: /home/nardor/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/blocking/shutdown.rs:51

This happens because reqwest (used under the hood) creates a runtime and drops it for every request.

[nardoor]

See the above PR for the step 1 fix, once rustic_core is updated, I'll update rustic.

[aawsome]

Actually the current webdav problem can be solved by just changing the implementation of WebDavFs - by using a spawned thread (std::thread::spawn) which calls all rustic_core functionality sync and then implement the async trait method by communicating with this thread using channels.

I made a POC implementation which needs quite some polish but already works.

[nardoor]

That's a good workaround! @aawsome