How to debug spurious deadlocks?

[milesj]

Hello, great library.

I built this experimental crate called starbase that utilizes scc under the hood but I'm encountering spurious deadlocks. This was originally using dashmap with no issue, but I wanted to try scc as it offered far more functionality.

I've done quite a bit of debugging and when it deadlocks, it happens in areas where drops (locks released?) should of happened, but are not. I can't seem to find anything wrong on my end. I've even tried forcing drops, and switching to async methods (hoping the timing would work better) with no luck: moonrepo/starbase#73

The deadlocks only occur in my project moon, but not in another project proto. Here's a PR where I'm trying to work around it: moonrepo/moon#1474

The starbase implementation is quite a lot, and so is moon. Trying to outline everything in this post would simply be too much. It would great if we could pair on this over chat, or discord, or something.

[wvwwvwwv]

Hi!

I just briefly saw the code (https://github.com/moonrepo/starbase/pull/73/files) and it seems like set_async is not used anywhere. If there is a piece of async code that invokes the synchronous set method, that may result in a deadlock. Therefore, I suggest that you find all the references of set in the code, and check each one of them whether it can be executed directly or indirectly in an asynchronous code block, or just remove synchronous methods completely.
=> Synchronous locking in async code is the most common pattern that leads to a deadlock.
=> If it is the case, deadlocks will be easily reproduced when you set the number of parallel workers to 1; you can just attach a debugger to the hanging process, and the one trying to acquire the lock on an entry will be just visible.

Given that you haven't encountered a deadlock with DashMap, I guess that the number of InstanceGuard instances is not that high, e.g., less than 100; if the number of entries in scc::HashMap is not high, the number of locks is most likely much smaller than that in DashMap, making it more susceptible to deadlocks.
=> The more the locks, the less chance you encounter a deadlock.
=> I'm pretty sure that the same deadlock will occur if you set the number of shards in DashMap to a small number or the number of InstanceGuard grows, because the locking behaviour of DashMap and scc::HashMap is identical.

In summary, I'd suggest that,

1. Get rid of the use of synchronous methods in asynchronous code.
2. In order to reproduce the issue, try using a single-threaded asynchronous runtime or set the number of workers to 1.
3. Once reproduced, attach a debugger to the process, and you'll find the culprit without difficulty.

[milesj]

Thanks for the quick response! I went ahead and converted both starbase and moon to use 100% async, but it did not solve the problem. I'll try and attach a debugger to see if I can figure out the root cause.

[wvwwvwwv]

Would you mind posting your findings here once you figure out the root cause? I’ll need to update the doc/examples.

[milesj]

Having no luck with the debugger. I'm not seeing anything out of the ordinary (nor anything scc related). Here's a screenshot of where the deadlock typically happens:

The main thread looks good, and all other threads are just tokio/parking_lot stuff. If I continue from the breakpoint, either the process completes, or the deadlock happens and I get this screen with no data.

And from here it just hangs indefinitely. I'm gonna punt this from now and move onto other things.

[milesj]

If you get the urge to mess with this yourself, you can do this:

• Clone moon: [email protected]:moonrepo/moon.git (branch 1.25-starbase)
• Clone starbase (relative to moon): [email protected]:moonrepo/starbase.git (branch async-core)
• Run cargo run -- check --all --log trace over and over to hit the deadlock

[wvwwvwwv]

That's interesting. I think there might have been a panic, but anything's possible. I'll look into it when I have time.

[milesj]

I decided to rewrite this crate and go with a different API (moonrepo/starbase#75), so this can probably be closed.

[wvwwvwwv]

I’ll then close this discussion!