Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOC-923 Update tasks to compute units for resource allocation #207

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

DOC-923 Update tasks to compute units for resource allocation #207

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bde5250
Update tasks to compute units
asimms41 Feb 20, 2025
2d86e2d
add related changes
asimms41 Feb 20, 2025
114b4e4
minor refinements
asimms41 Feb 20, 2025
0ea628d
Address review comments
asimms41 Feb 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions modules/develop/pages/connect/configuration/resource-management.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
= Manage Pipeline Resources on BYOC and Dedicated Clusters
= Manage Pipeline Resources
:description: Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance.
:page-aliases: develop:connect/configuration/scale-pipelines.adoc

{description}

== Prerequisites

- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated cluster]
- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated], xref:get-started:cluster-types/serverless-pro.adoc[Serverless Pro], or xref:get-started:cluster-types/serverless.adoc[Serverless Standard] cluster.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From our production env it looks like you can now scale pipeline resources for all cluster types.

Copy link
Member

@nicolaferraro nicolaferraro Feb 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think serverless still keeps the value at 1, but the constraint will likely be removed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried both Serverless types in our test environments and they both seemed to scale (in terms of tasks/compute units). I guess I just need to know what you are going to release.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nicolaferraro any reason why we cannot remove this restriction and allow Serverless user to configure up to 16 compute units (i.e. 1.6 cores and one instance)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've assumed that Serverless (Standard and Pro) will allow pipeline scaling.

- An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the xref:redpanda-connect:components:processors/benchmark.adoc[`benchmark` processor].

=== Understanding tasks
=== Understanding compute units

A task is a unit of computation that allocates a specific amount of CPU and memory to a data pipeline to handle message throughput. By default, each pipeline is allocated one task, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of approximately 1 MB/sec. You can allocate up to a maximum of 18 tasks per pipeline.
A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of up to 1 MB/sec.

Server resources are charged at an hourly rate in compute unit hours and you can allocate up to a maximum of 18 compute units per pipeline.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can say that the maximum number of compute units varies by provider, it's no longer 18 after we did some tests.
It's 17 on AWS, 15 on GCP and 16 on Azure (don't think we need to state that explicitly).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jrkinley - should we state the compute unit limit per provider or remove all mention of an upper limit?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nicolaferraro wouldn't it be easier to set the same limit across all providers? and take the lowest one of 15? Then we can say 75% of the server resource is available for scheduling pipelines and 25% is reserved for system overhead (i.e. Kubernetes tax).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asimms41 I'd caveat "up to 1MB/s" as every pipeline can and probably will be wildly different. See my description here: What-is-a-Compute-Unit?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are currently increasing the maximum pipeline size, so that twice and 4 times as large pipelines will be able to run. The compute units for the larger pipelines are:

  • 36 and 76 for AWS
  • 33 and 72 for GCP
  • 35 and 74 for Azure
    Maybe we can say what the limit is more or less and that it depends on the provider? Not sure whether 15 vs 17 makes so much difference for users. The UI is going to validate in anyway.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find it strange for the limits to be different. What was so different in testing to warrant a 200 millicore difference between providers?

Will all pipelines this large run on the 8-core instance sizes? i.e. we're not planning to use 4-core instance sizes for the 33-36 compute unit pipelines?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we provision 4-core and 8-core instances in addition to the initial 2-core ones, so that we don't waste too much

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we agreed with Nicola to go with the unified numbers across providers, which are:

  • 15 compute units for 2-core machine
  • 33 to run on 4-core
  • 72 for 8-core

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that makes sense. Thanks @tomasz-sadura @nicolaferraro.

Copy link
Contributor Author

@asimms41 asimms41 Feb 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated max to 15 until limits for larger pipelines are configurable.


|===
| Number of Tasks | CPU | Memory
| Number of compute units | CPU | Memory

| 1
| 0.1 CPU (`100m`)
Expand Down Expand Up @@ -90,11 +92,11 @@ A task is a unit of computation that allocates a specific amount of CPU and memo

|===

NOTE: For pipelines with embedded Ollama AI components, one GPU task is automatically allocated to the pipeline, which is equivalent to 30 tasks or 3.0 CPU (`3000m`) and 12 GB of memory (`12000M`).
NOTE: For pipelines with embedded Ollama AI components, one GPU is automatically allocated to the pipeline, which is equivalent to 30 compute units, or 3.0 CPU (`3000m`) and 12 GB of memory (`12000M`).

=== Set an initial resource limit

When you create a data pipeline, you can allocate a fixed amount of compute resources to it using tasks.
When you create a data pipeline, you can allocate a fixed amount of server resources to it using compute units.

[NOTE]
====
Expand All @@ -109,12 +111,12 @@ To set an initial resource limit:
. Select the **Redpanda Connect** tab.
. Click **Create pipeline**.
. Enter details for your pipeline, including a short name and description.
. In the **Tasks** box, leave the default **1** task to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate up to a maximum of 18 tasks.
. In the **Compute units** box, leave the default **1** compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate up to a maximum of 18 compute units.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above for the max

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated max to 15 until limits for larger pipelines are configurable.

. Add your pipeline configuration and click **Create** to run it.

=== Scale resources

View the compute resources allocated to a data pipeline, and manually scale those resources to improve performance or decrease resource consumption.
View the server resources allocated to a data pipeline, and manually scale those resources to improve performance or decrease resource consumption.

To view resources already allocated to a data pipeline:

Expand All @@ -127,8 +129,8 @@ Cloud UI::
. Go to the cluster where the pipeline is set up.
. On the **Connect** page, select your pipeline and look at the value for **Resources**.
+
* CPU resources are displayed first, in milliCPU. For example, `1` task is `100m` or 0.1 CPU.
* Memory is displayed next in megabytes. For example, `1` task is `400M` or 400 MB.
* CPU resources are displayed first, in milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU.
* Memory is displayed next in megabytes. For example, `1` compute unit is `400M` or 400 MB.

--
Data Plane API::
Expand All @@ -137,8 +139,8 @@ Data Plane API::
. xref:manage:api/cloud-api-quickstart.adoc#try-the-cloud-api[Authenticate and get the base URL] for the Data Plane API.
. Make a request to xref:api:ROOT:cloud-dataplane-api.adoc#get-/v1alpha2/redpanda-connect/pipelines[`GET /v1alpha2/redpanda-connect/pipelines`], which lists details of all pipelines on your cluster by ID.
+
* Memory (`memory_shares`) is displayed in megabytes. For example, `1` task is `400M` or 400 MB.
* CPU resources (`cpu_shares`) are displayed milliCPU. For example, `1` task is `100m` or 0.1 CPU.
* Memory (`memory_shares`) is displayed in megabytes. For example, `1` compute unit is `400M` or 400 MB.
* CPU resources (`cpu_shares`) are displayed milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU.

--
=====
Expand All @@ -153,7 +155,7 @@ Cloud UI::
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Go to the cluster where the pipeline is set up.
. On the **Connect** page, select your pipeline and click **Edit**.
. In the **Tasks** box, update the number of tasks. One task provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 tasks per pipeline.
. In the **Compute units** box, update the number of compute units. One compute unit provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 compute units per pipeline.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't see the updates to the UI yet in our prod or integration.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above for max. Will make sure the UI is up to date

Copy link
Contributor Author

@asimms41 asimms41 Feb 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated max to 15 as in thread below until limits for larger pipelines are configurable. Although I could add up to 90 tasks through the UI?

. Click **Update** to apply your changes. The specified resources are available immediately.

--
Expand Down
4 changes: 2 additions & 2 deletions modules/develop/pages/connect/connect-quickstart.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ All Redpanda Connect configurations use a YAML file split into three sections:

. Go to the **Connect** page on your cluster and click **Create pipeline**.
. In **Pipeline name**, enter **emailprocessor-pipeline** and add a short description. For example, **Transforms email data using a mutation processor**.
. In the **Tasks** box, leave the default value of **1**. Tasks are used to allocate resources to a pipeline. One task is equivalent to 0.1 CPU and 400 MB of memory, and provides a message throughput of approximately 1 MB/sec.
. In the **Compute units** box, leave the default value of **1**. Compute units are used to allocate server resources to a pipeline. One compute unit is equivalent to 0.1 CPU and 400 MB of memory, and provides a message throughput of approximately 1 MB/sec.
. In the **Configuration** box, paste the following configuration.

+
Expand Down Expand Up @@ -249,5 +249,5 @@ When you've finished experimenting with your data pipeline, you can delete the p
* Choose xref:develop:connect/components/catalog.adoc[connectors for your use case].
* Learn how to xref:develop:connect/configuration/secret-management.adoc[add secrets to your pipeline].
* Learn how to xref:develop:connect/configuration/monitor-connect.adoc[monitor a data pipeline on a BYOC or Dedicated cluster].
* Learn how to xref:develop:connect/configuration/scale-pipelines.adoc[manually scale resources for a pipeline on a BYOC or Dedicated cluster].
* Learn how to xref:develop:connect/configuration/scale-pipelines.adoc[manually scale resources for a pipeline].
* Learn how to xref:redpanda-connect:guides:getting_started.adoc[configure, test, and run a data pipeline locally].