DOC-923 Update tasks to compute units for resource allocation #207
Conversation
✅ Deploy Preview for rp-cloud ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
| :description: Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance. | ||
| :page-aliases: develop:connect/configuration/scale-pipelines.adoc | ||
|
|
||
| {description} | ||
|
|
||
| == Prerequisites | ||
|
|
||
| - A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated cluster] | ||
| - A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated], xref:get-started:cluster-types/serverless-pro.adoc[Serverless Pro], or xref:get-started:cluster-types/serverless.adoc[Serverless Standard] cluster. |
There was a problem hiding this comment.
From our production env it looks like you can now scale pipeline resources for all cluster types.
There was a problem hiding this comment.
I think serverless still keeps the value at 1, but the constraint will likely be removed
There was a problem hiding this comment.
I tried both Serverless types in our test environments and they both seemed to scale (in terms of tasks/compute units). I guess I just need to know what you are going to release.
There was a problem hiding this comment.
@nicolaferraro any reason why we cannot remove this restriction and allow Serverless user to configure up to 16 compute units (i.e. 1.6 cores and one instance)?
There was a problem hiding this comment.
I've assumed that Serverless (Standard and Pro) will allow pipeline scaling.
| @@ -153,7 +155,7 @@ Cloud UI:: | |||
| . Log in to https://cloud.redpanda.com[Redpanda Cloud^]. | |||
| . Go to the cluster where the pipeline is set up. | |||
| . On the **Connect** page, select your pipeline and click **Edit**. | |||
| . In the **Tasks** box, update the number of tasks. One task provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 tasks per pipeline. | |||
| . In the **Compute units** box, update the number of compute units. One compute unit provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 compute units per pipeline. | |||
There was a problem hiding this comment.
I can't see the updates to the UI yet in our prod or integration.
There was a problem hiding this comment.
See above for max. Will make sure the UI is up to date
There was a problem hiding this comment.
Updated max to 15 as in thread below until limits for larger pipelines are configurable. Although I could add up to 90 tasks through the UI?
| A task is a unit of computation that allocates a specific amount of CPU and memory to a data pipeline to handle message throughput. By default, each pipeline is allocated one task, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of approximately 1 MB/sec. You can allocate up to a maximum of 18 tasks per pipeline. | ||
| A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of up to 1 MB/sec. | ||
|
|
||
| Server resources are charged at an hourly rate in compute unit hours and you can allocate up to a maximum of 18 compute units per pipeline. |
There was a problem hiding this comment.
Maybe we can say that the maximum number of compute units varies by provider, it's no longer 18 after we did some tests.
It's 17 on AWS, 15 on GCP and 16 on Azure (don't think we need to state that explicitly).
There was a problem hiding this comment.
@jrkinley - should we state the compute unit limit per provider or remove all mention of an upper limit?
There was a problem hiding this comment.
@nicolaferraro wouldn't it be easier to set the same limit across all providers? and take the lowest one of 15? Then we can say 75% of the server resource is available for scheduling pipelines and 25% is reserved for system overhead (i.e. Kubernetes tax).
There was a problem hiding this comment.
@asimms41 I'd caveat "up to 1MB/s" as every pipeline can and probably will be wildly different. See my description here: What-is-a-Compute-Unit?
There was a problem hiding this comment.
We are currently increasing the maximum pipeline size, so that twice and 4 times as large pipelines will be able to run. The compute units for the larger pipelines are:
- 36 and 76 for AWS
- 33 and 72 for GCP
- 35 and 74 for Azure
Maybe we can say what the limit is more or less and that it depends on the provider? Not sure whether 15 vs 17 makes so much difference for users. The UI is going to validate in anyway.
There was a problem hiding this comment.
I find it strange for the limits to be different. What was so different in testing to warrant a 200 millicore difference between providers?
Will all pipelines this large run on the 8-core instance sizes? i.e. we're not planning to use 4-core instance sizes for the 33-36 compute unit pipelines?
There was a problem hiding this comment.
yes, we provision 4-core and 8-core instances in addition to the initial 2-core ones, so that we don't waste too much
There was a problem hiding this comment.
we agreed with Nicola to go with the unified numbers across providers, which are:
- 15 compute units for 2-core machine
- 33 to run on 4-core
- 72 for 8-core
There was a problem hiding this comment.
I think that makes sense. Thanks @tomasz-sadura @nicolaferraro.
There was a problem hiding this comment.
Updated max to 15 until limits for larger pipelines are configurable.
| @@ -109,12 +111,12 @@ To set an initial resource limit: | |||
| . Select the **Redpanda Connect** tab. | |||
| . Click **Create pipeline**. | |||
| . Enter details for your pipeline, including a short name and description. | |||
| . In the **Tasks** box, leave the default **1** task to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate up to a maximum of 18 tasks. | |||
| . In the **Compute units** box, leave the default **1** compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate up to a maximum of 18 compute units. | |||
There was a problem hiding this comment.
Updated max to 15 until limits for larger pipelines are configurable.
| @@ -153,7 +155,7 @@ Cloud UI:: | |||
| . Log in to https://cloud.redpanda.com[Redpanda Cloud^]. | |||
| . Go to the cluster where the pipeline is set up. | |||
| . On the **Connect** page, select your pipeline and click **Edit**. | |||
| . In the **Tasks** box, update the number of tasks. One task provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 tasks per pipeline. | |||
| . In the **Compute units** box, update the number of compute units. One compute unit provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 compute units per pipeline. | |||
There was a problem hiding this comment.
See above for max. Will make sure the UI is up to date
| :description: Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance. | ||
| :page-aliases: develop:connect/configuration/scale-pipelines.adoc | ||
|
|
||
| {description} | ||
|
|
||
| == Prerequisites | ||
|
|
||
| - A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated cluster] | ||
| - A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated], xref:get-started:cluster-types/serverless-pro.adoc[Serverless Pro], or xref:get-started:cluster-types/serverless.adoc[Serverless Standard] cluster. |
There was a problem hiding this comment.
@nicolaferraro any reason why we cannot remove this restriction and allow Serverless user to configure up to 16 compute units (i.e. 1.6 cores and one instance)?
| A task is a unit of computation that allocates a specific amount of CPU and memory to a data pipeline to handle message throughput. By default, each pipeline is allocated one task, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of approximately 1 MB/sec. You can allocate up to a maximum of 18 tasks per pipeline. | ||
| A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of up to 1 MB/sec. | ||
|
|
||
| Server resources are charged at an hourly rate in compute unit hours and you can allocate up to a maximum of 18 compute units per pipeline. |
There was a problem hiding this comment.
@nicolaferraro wouldn't it be easier to set the same limit across all providers? and take the lowest one of 15? Then we can say 75% of the server resource is available for scheduling pipelines and 25% is reserved for system overhead (i.e. Kubernetes tax).
| A task is a unit of computation that allocates a specific amount of CPU and memory to a data pipeline to handle message throughput. By default, each pipeline is allocated one task, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of approximately 1 MB/sec. You can allocate up to a maximum of 18 tasks per pipeline. | ||
| A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of up to 1 MB/sec. | ||
|
|
||
| Server resources are charged at an hourly rate in compute unit hours and you can allocate up to a maximum of 18 compute units per pipeline. |
There was a problem hiding this comment.
@asimms41 I'd caveat "up to 1MB/s" as every pipeline can and probably will be wildly different. See my description here: What-is-a-Compute-Unit?
Description
Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline: 24th February
Page previews
Manage Pipeline Resources
Quickstart
This pull request includes changes to the documentation for managing pipeline resources. The changes focus on updating terminology and refining instructions to improve clarity and accuracy.
Key changes include:
Terminology updates:
modules/develop/pages/connect/configuration/resource-management.adoc, [1] [2] [3] [4] [5] [6]modules/develop/pages/connect/configuration/resource-management.adoc, [1] [2]Instruction refinements:
modules/develop/pages/connect/configuration/resource-management.adoc, modules/develop/pages/connect/configuration/resource-management.adocL1-R21)modules/develop/pages/connect/configuration/resource-management.adoc, [1] [2]Cross-references and examples:
modules/develop/pages/connect/connect-quickstart.adoc, [1] [2]Checks