feat!: multiple mutexes and semaphores

Fixes #5022
Fixes #11859

This is mostly backwards compatbile. It is minorly breaking only in as
much as semaphore + mutex at the same time will now require both,
whereas before it ignored the mutex.

As a workflows users I would like to be able to use more than one sync
option in a workflow or template.

Added `mutexes` and `semaphores` to the `synchronization` block in
workflow types as lists. Legacy `mutex` and `semaphore` kept in and
continue to work (will be added to the mutexes/semaphores list) but
marked for deprecation.

Logic changed to acquire a full list of all synchronization
items (`syncItem`) when `TryAcquire` and `Release` from sync manager.

`TryAcquire` from sync_manager` calls a new `checkAcquire` method,
whilst holding the sync_manager internal golang `sync.Mutex` to check
if they are all acquirable before acquiring them. This honors
priority.

`TryAcquire` also returns the failed lock name in the case of failure
to acquire to reduce the number of exported `package sync` methods and
complexity outside of `sync`.

The interface to the workflow status block has not changed.

Reduced the exported types and methods from the sync module as far as
possible (lower case instead of upper case).

Removed the internal golang sync mutex from `PrioritySemaphore` as all
methods are called from `sync_manager.go` when under it's own
lock. The alternative was unnecessarily complicating the calls inside
semaphore.go. This is faster, but less clean in case someone wants to
reuse the PrioritySemaphore type.

Renamed the manager in `sync_manager.go` where used: `cm`->`sm` and
`concurrencyManager`->`syncManager`.

New and updated unit tests.
New and updated e2e smoke tests.

Signed-off-by: Alan Clucas <[email protected]>

docs/executor_swagger.md

@@ -3534,7 +3534,9 @@ otherwise).
 | Name | Type | Go type | Required | Default | Description | Example |
 |------|------|---------|:--------:| ------- |-------------|---------|
 | mutex | [Mutex](#mutex)| `Mutex` |  | |  |  |
+| mutexes | [][Mutex](#mutex)| `[]*Mutex` |  | | Mutexes holds the list of Mutex lock details |  |
 | semaphore | [SemaphoreRef](#semaphore-ref)| `SemaphoreRef` |  | |  |  |
+| semaphores | [][SemaphoreRef](#semaphore-ref)| `[]*SemaphoreRef` |  | | Semaphores holds the list of Semaphores configuration |  |
 
 
 

docs/fields.md

@@ -1632,8 +1632,10 @@ Synchronization holds synchronization lock configuration
 ### Fields
 | Field Name | Field Type | Description   |
 |:----------:|:----------:|---------------|
-|`mutex`|[`Mutex`](#mutex)|Mutex holds the Mutex lock details|
-|`semaphore`|[`SemaphoreRef`](#semaphoreref)|Semaphore holds the Semaphore configuration|
+|`mutex`|[`Mutex`](#mutex)|Mutex holds the Mutex lock details - deprecated, use mutexes instead|
+|`mutexes`|`Array<`[`Mutex`](#mutex)`>`|Mutexes holds the list of Mutex lock details|
+|`semaphore`|[`SemaphoreRef`](#semaphoreref)|Semaphore holds the Semaphore configuration - deprecated, use semaphores instead|
+|`semaphores`|`Array<`[`SemaphoreRef`](#semaphoreref)`>`|Semaphores holds the list of Semaphores configuration|
 
 ## Template
 
@@ -2349,14 +2351,6 @@ Backoff is a backoff strategy to use within retryStrategy
 
 Mutex holds Mutex configuration
 
-<details markdown>
-<summary>Examples with this field (click to open)</summary>
-
-- [`synchronization-mutex-tmpl-level.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/synchronization-mutex-tmpl-level.yaml)
-
-- [`synchronization-mutex-wf-level.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/synchronization-mutex-wf-level.yaml)
-</details>
-
 ### Fields
 | Field Name | Field Type | Description   |
 |:----------:|:----------:|---------------|
@@ -3231,14 +3225,6 @@ NodeSynchronizationStatus stores the status of a node
 
 MutexStatus contains which objects hold mutex locks, and which objects this workflow is waiting on to release locks.
 
-<details markdown>
-<summary>Examples with this field (click to open)</summary>
-
-- [`synchronization-mutex-tmpl-level.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/synchronization-mutex-tmpl-level.yaml)
-
-- [`synchronization-mutex-wf-level.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/synchronization-mutex-wf-level.yaml)
-</details>
-
 ### Fields
 | Field Name | Field Type | Description   |
 |:----------:|:----------:|---------------|

docs/synchronization.md

@@ -1,6 +1,7 @@
 # Synchronization
 
 > v2.10 and after
+> v3.6 for multiple
 
 ## Introduction
 
@@ -43,10 +44,10 @@ metadata:
 spec:
   entrypoint: whalesay
   synchronization:
-    semaphore:
-      configMapKeyRef:
-        name: my-config
-        key: workflow
+    semaphores:
+      - configMapKeyRef:
+          name: my-config
+          key: workflow
   templates:
   - name: whalesay
     container:
@@ -65,8 +66,8 @@ metadata:
 spec:
   entrypoint: whalesay
   synchronization:
-    mutex:
-      name: workflow
+    mutexes:
+      - name: workflow
   templates:
   - name: whalesay
     container:
@@ -105,10 +106,10 @@ spec:
 
   - name: acquire-lock
     synchronization:
-      semaphore:
-        configMapKeyRef:
-          name: my-config
-          key: template
+      semaphores:
+        - configMapKeyRef:
+            name: my-config
+            key: template
     container:
       image: alpine:latest
       command: [sh, -c]
@@ -137,8 +138,8 @@ spec:
 
   - name: acquire-lock
     synchronization:
-      mutex:
-        name: template
+      mutexes:
+        - name: template
     container:
       image: alpine:latest
       command: [sh, -c]
@@ -152,6 +153,26 @@ Examples:
 1. [Step level semaphore](https://github.com/argoproj/argo-workflows/blob/main/examples/synchronization-tmpl-level.yaml)
 1. [Step level mutex](https://github.com/argoproj/argo-workflows/blob/main/examples/synchronization-mutex-tmpl-level.yaml)
 
+### Multiple locks
+
+You can specify multiple locks in a single workflow or template.
+
+```yaml
+synchronization:
+  mutexes:
+    - name: alpha
+    - name: beta
+  semaphores:
+    - configMapKeyRef:
+        key: foo
+        name: my-config
+    - configMapKeyRef:
+        key: bar
+        name: my-config
+```
+
+The workflow will block until all of these locks are available.
+
 ### Queuing
 
 When a Workflow cannot take a lock it will be placed into a ordered queue.
@@ -162,6 +183,33 @@ The queue is then ordered by `CreationTimestamp` of the Workflow; older Workflow
 
 Workflows are only be allowed to take a lock if they are at the front of the queue for that lock.
 
+!!! Warning
+    If a Workflow is at the front of the queue and it needs to acquire multiple locks, all other Workflows that also need those same locks will wait. This applies even if the other Workflows only wish to acquire a subset of those locks.
+
+### Legacy
+
+In workflows prior to 3.6 you can only specify one lock in any one workflow or template using either a mutex:
+
+```yaml
+    synchronization:
+      mutex:
+     ...
+```
+
+or a semaphore:
+
+```yaml
+    synchronizaion:
+   semamphore:
+     ...
+```
+
+Specifying both would not work in <3.6, only the semaphore would be used.
+
+The single `mutex` and `semaphore` syntax still works in version 3.6 but is considered deprecated.
+Both the `mutex` and the `semaphore` will be taken in version 3.6 with this syntax.
+This syntax can be mixed with `mutexes` and `semaphores`, all locks will be required.
+
 ## Parallelism
 
 See also [how you can restrict parallelism](./parallelism.md) in other ways.

examples/synchronization-mutex-tmpl-level.yaml

@@ -28,18 +28,18 @@ spec:
 
   - name: acquire-lock
     synchronization:
-      mutex:
-        name: welcome
+      mutexes:
+        - name: welcome
     container:
       image: alpine:latest
       command: [sh, -c]
       args: ["sleep 20; echo acquired lock"]
 
   - name: acquire-lock-1
     synchronization:
-      mutex:
-        name: test
+      mutexes:
+        - name: test
     container:
       image: alpine:latest
       command: [sh, -c]
-      args: ["sleep 50; echo acquired lock"]
+      args: ["sleep 50; echo acquired lock"]

examples/synchronization-mutex-wf-level.yaml

@@ -7,8 +7,8 @@ metadata:
 spec:
   entrypoint: whalesay
   synchronization:
-    mutex:
-      name:  test
+    mutexes:
+      - name:  test
   templates:
     - name: whalesay
       container:

examples/synchronization-tmpl-level.yaml

@@ -27,10 +27,10 @@ spec:
 
   - name: acquire-lock
     synchronization:
-      semaphore:
-        configMapKeyRef:
-          name: my-config
-          key: template
+      semaphores:
+        - configMapKeyRef:
+            name: my-config
+            key: template
     container:
       image: alpine:latest
       command: [sh, -c]

examples/synchronization-wf-level.yaml

@@ -1,4 +1,4 @@
-# This example demonstrates the use of a Synchronization lock on workflow execution. Synchronization lock limits 
+# This example demonstrates the use of a Synchronization lock on workflow execution. Synchronization lock limits
 # the number of concurrent workflow execution in the namespace which has same Synchronization lock. Synchronization
 # limit value can be configured in configmap.
 # Eg.:
@@ -16,10 +16,10 @@ metadata:
 spec:
   entrypoint: whalesay
   synchronization:
-    semaphore:
-      configMapKeyRef:
-        name: my-config
-        key: workflow
+    semaphores:
+      - configMapKeyRef:
+          name: my-config
+          key: workflow
   templates:
     - name: whalesay
       container: