Skip to content

Commit

Permalink
Feature/10215 prevent deadlocks (#153)
Browse files Browse the repository at this point in the history
Added new section for "Prevent Deadlocks" in "Working With Semaphores"
  • Loading branch information
cortex-mtr authored Aug 15, 2024
1 parent 719b4de commit b5b3082
Showing 1 changed file with 21 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,27 @@ If a block tries to use a semaphore that already exists (using the same [Scope][

### Preventing Deadlocks

TODO
Deadlocks in flows can occur when multiple executions have acquired different semaphores, and then each execution wants to acquire another semaphore that is being used by another execution, which in turn is waiting for first execution to release its semaphore. E.g.

```
There exists two semaphores:
- Semaphore A with a ConcurrencyLimit of 1 and a queue
- Semaphore B with a ConcurrencyLimit of 1 and a queue
There exists two executions:
- Execution A acquires Semaphore A
- Execution B acquires Semaphore B
- Execution A tries to acquire Semaphore B and joins the queue
- Execution B tries to acquire Semaphore A and joins the queue
Both Execution A and Execution B are deadlocked.
```

In this example, Execution A can't proceed as it's waiting for the semaphore that Execution B is using; which in turn is waiting for the semaphore that Execution A is using.

The best way to avoid these deadlocks is to design your flows to avoid them happening in the first place. In the example above, if both executions will need both semaphores to operate, a change that could be made would be to combine both semaphores into one.

Another way to mitigate the problem would be to use the [QueueTimeout][] property when defining the [QueueSettings][] for each semaphore. Setting the [QueueTimeout][] to a duration reasonably higher than the expected execution time of the operations inside of the semaphore would allow the execution to throw if it has been waiting too long, likely due to a deadlock having occurred. The [SemaphoreCouldNotBeAcquiredException][QueueTimeoutReached] thrown could then be caught to allow the execution to release its semaphores and try again. In the example above, if Semaphore A had a [QueueTimeout][], Execution B timeouts out as it waits to acquire Semaphore A and releases Semaphore B. Execution A then acquires Semaphore B. Execution B retries to acquire Semaphore B and joins the queue. Execution A then finishes its operations, releasing both semaphores and allows Execution B to complete its operations as well.

### Known Limitations

Expand Down

0 comments on commit b5b3082

Please sign in to comment.