-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: synchronization and parallelism improvements
Split from #13358 This is a rewrite of the docs for synchronization and parallelism, split from the multiple mutexes and semaphores code. This PR is suitable for backporting to 3.5 and 3.4. * New markdown document on parallelism: - locks and parallelism will generally have different use cases. I think it is better not to have them on the same page so you don't think you could use one instead of the other - As far as I know more pages doesn't cost us anything. - priority for parallelism documented - paralellism in workflow and template spec better explained * Synchronisation adds: - information on queuing - better explain whay a mutex is - explain behaviour in case of specifying both semaphore and mutex Conform to style guide and 1 sentence per line Signed-off-by: Alan Clucas <[email protected]>
- Loading branch information
Showing
4 changed files
with
77 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -185,6 +185,7 @@ memoizing | |
metadata | ||
minikube | ||
mutex | ||
mutexes | ||
namespace | ||
namespaces | ||
natively | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
# Limiting parallelism | ||
|
||
You can restrict the number of workflows being executed at any time using these mechanisms. | ||
|
||
## Controller level | ||
|
||
You can limit the total number of workflows that can execute at any one time in the [workflow controller ConfigMap](./workflow-controller-configmap.yaml). | ||
|
||
```yaml | ||
data: | ||
parallelism: "10" | ||
``` | ||
You can also limit the number of workflows that can execute in a single namespace. | ||
```yaml | ||
data: | ||
namespaceParallelism: "4" | ||
``` | ||
Workflows that are executing but restricted from running more nodes due to the other mechanisms will still count towards the parallelism limits. | ||
### Priority | ||
Workflows can have a `priority` set in their specification. | ||
|
||
Workflows with a higher priority number that have not started due to controller level parallelism will be started before lower priority workflows. | ||
|
||
## Workflow level | ||
|
||
You can restrict parallelism within a workflow using `parallelism` within a workflow or template. | ||
This only restricts total concurrent executions of steps or tasks within the same workflow. | ||
|
||
Examples | ||
|
||
1 [`parallelism-limit.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/parallelism-limit.yaml) restricts the parallelism of a [loop](./walk-through/loops.md) | ||
1 [`parallelism-nested.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/parallelism-nested.yaml) restricts the parallelism of a nested loop | ||
1 [`parallelism-nested-dag.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/parallelism-nested-dag.yaml) restricts the number of dag tasks that can be run at any one time | ||
1 [`parallelism-nested-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/parallelism-nested-workflow.yaml) shows how parallelism is inherited by children | ||
1 [`parallelism-template-limit.yaml`](https://github.com/argoproj/argo-workflows/blob/main/examples/parallelism-template-limit.yaml) shows how parallelism of looped templates is also restricted | ||
|
||
## Synchronization | ||
|
||
You can use [mutexes and semaphores](./synchronization.md) to control the parallel execution of sections of a workflow. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters