From bde5250ab4ebbdb1ee211286c9f0355a72d86b35 Mon Sep 17 00:00:00 2001 From: Angela Simms Date: Thu, 20 Feb 2025 12:53:01 +0000 Subject: [PATCH 1/4] Update tasks to compute units --- .../configuration/resource-management.adoc | 26 ++++++++++--------- 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/modules/develop/pages/connect/configuration/resource-management.adoc b/modules/develop/pages/connect/configuration/resource-management.adoc index 7ad6a9cd..c4e0643c 100644 --- a/modules/develop/pages/connect/configuration/resource-management.adoc +++ b/modules/develop/pages/connect/configuration/resource-management.adoc @@ -9,12 +9,14 @@ - A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated cluster] - An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the xref:redpanda-connect:components:processors/benchmark.adoc[`benchmark` processor]. -=== Understanding tasks +=== Understanding compute units -A task is a unit of computation that allocates a specific amount of CPU and memory to a data pipeline to handle message throughput. By default, each pipeline is allocated one task, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of approximately 1 MB/sec. You can allocate up to a maximum of 18 tasks per pipeline. +A compute unit allocates a specific amount of server resource (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of up to 1 MB/sec. You can allocate up to a maximum of 18 compute units per pipeline. + +Note: Server resources are charged at an hourly rate in compute unit hours. |=== -| Number of Tasks | CPU | Memory +| Number of compute units | CPU | Memory | 1 | 0.1 CPU (`100m`) @@ -90,11 +92,11 @@ A task is a unit of computation that allocates a specific amount of CPU and memo |=== -NOTE: For pipelines with embedded Ollama AI components, one GPU task is automatically allocated to the pipeline, which is equivalent to 30 tasks or 3.0 CPU (`3000m`) and 12 GB of memory (`12000M`). +NOTE: For pipelines with embedded Ollama AI components, one GPU is automatically allocated to the pipeline, which is equivalent to 30 compute units or 3.0 CPU (`3000m`) and 12 GB of memory (`12000M`). === Set an initial resource limit -When you create a data pipeline, you can allocate a fixed amount of compute resources to it using tasks. +When you create a data pipeline, you can allocate a fixed amount of server resources to it using compute units. [NOTE] ==== @@ -109,12 +111,12 @@ To set an initial resource limit: . Select the **Redpanda Connect** tab. . Click **Create pipeline**. . Enter details for your pipeline, including a short name and description. -. In the **Tasks** box, leave the default **1** task to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate up to a maximum of 18 tasks. +. In the **Compute units** box, leave the default **1** compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate up to a maximum of 18 compute units. . Add your pipeline configuration and click **Create** to run it. === Scale resources -View the compute resources allocated to a data pipeline, and manually scale those resources to improve performance or decrease resource consumption. +View the server resources allocated to a data pipeline, and manually scale those resources to improve performance or decrease resource consumption. To view resources already allocated to a data pipeline: @@ -127,8 +129,8 @@ Cloud UI:: . Go to the cluster where the pipeline is set up. . On the **Connect** page, select your pipeline and look at the value for **Resources**. + -* CPU resources are displayed first, in milliCPU. For example, `1` task is `100m` or 0.1 CPU. -* Memory is displayed next in megabytes. For example, `1` task is `400M` or 400 MB. +* CPU resources are displayed first, in milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU. +* Memory is displayed next in megabytes. For example, `1` compute unit is `400M` or 400 MB. -- Data Plane API:: @@ -137,8 +139,8 @@ Data Plane API:: . xref:manage:api/cloud-api-quickstart.adoc#try-the-cloud-api[Authenticate and get the base URL] for the Data Plane API. . Make a request to xref:api:ROOT:cloud-dataplane-api.adoc#get-/v1alpha2/redpanda-connect/pipelines[`GET /v1alpha2/redpanda-connect/pipelines`], which lists details of all pipelines on your cluster by ID. + -* Memory (`memory_shares`) is displayed in megabytes. For example, `1` task is `400M` or 400 MB. -* CPU resources (`cpu_shares`) are displayed milliCPU. For example, `1` task is `100m` or 0.1 CPU. +* Memory (`memory_shares`) is displayed in megabytes. For example, `1` compute unit is `400M` or 400 MB. +* CPU resources (`cpu_shares`) are displayed milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU. -- ===== @@ -153,7 +155,7 @@ Cloud UI:: . Log in to https://cloud.redpanda.com[Redpanda Cloud^]. . Go to the cluster where the pipeline is set up. . On the **Connect** page, select your pipeline and click **Edit**. -. In the **Tasks** box, update the number of tasks. One task provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 tasks per pipeline. +. In the **Compute units** box, update the number of compute units. One task provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 compute units per pipeline. . Click **Update** to apply your changes. The specified resources are available immediately. -- From 2d86e2d1b0c72f03288efca8fc87c9596ced6c19 Mon Sep 17 00:00:00 2001 From: Angela Simms Date: Thu, 20 Feb 2025 14:16:39 +0000 Subject: [PATCH 2/4] add related changes --- .../pages/connect/configuration/resource-management.adoc | 6 +++--- modules/develop/pages/connect/connect-quickstart.adoc | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/modules/develop/pages/connect/configuration/resource-management.adoc b/modules/develop/pages/connect/configuration/resource-management.adoc index c4e0643c..3efbccb7 100644 --- a/modules/develop/pages/connect/configuration/resource-management.adoc +++ b/modules/develop/pages/connect/configuration/resource-management.adoc @@ -1,4 +1,4 @@ -= Manage Pipeline Resources on BYOC and Dedicated Clusters += Manage Pipeline Resources :description: Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance. :page-aliases: develop:connect/configuration/scale-pipelines.adoc @@ -6,7 +6,7 @@ == Prerequisites -- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated cluster] +- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated], xref:get-started:cluster-types/serverless-pro.adoc[Serverless Pro], or xref:get-started:cluster-types/serverless.adoc[Serverless Standard] cluster - An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the xref:redpanda-connect:components:processors/benchmark.adoc[`benchmark` processor]. === Understanding compute units @@ -155,7 +155,7 @@ Cloud UI:: . Log in to https://cloud.redpanda.com[Redpanda Cloud^]. . Go to the cluster where the pipeline is set up. . On the **Connect** page, select your pipeline and click **Edit**. -. In the **Compute units** box, update the number of compute units. One task provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 compute units per pipeline. +. In the **Compute units** box, update the number of compute units. One compute unit provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 compute units per pipeline. . Click **Update** to apply your changes. The specified resources are available immediately. -- diff --git a/modules/develop/pages/connect/connect-quickstart.adoc b/modules/develop/pages/connect/connect-quickstart.adoc index fb4623e9..50168c22 100644 --- a/modules/develop/pages/connect/connect-quickstart.adoc +++ b/modules/develop/pages/connect/connect-quickstart.adoc @@ -95,7 +95,7 @@ All Redpanda Connect configurations use a YAML file split into three sections: . Go to the **Connect** page on your cluster and click **Create pipeline**. . In **Pipeline name**, enter **emailprocessor-pipeline** and add a short description. For example, **Transforms email data using a mutation processor**. -. In the **Tasks** box, leave the default value of **1**. Tasks are used to allocate resources to a pipeline. One task is equivalent to 0.1 CPU and 400 MB of memory, and provides a message throughput of approximately 1 MB/sec. +. In the **Compute units** box, leave the default value of **1**. Compute units are used to allocate server resources to a pipeline. One compute unit is equivalent to 0.1 CPU and 400 MB of memory, and provides a message throughput of approximately 1 MB/sec. . In the **Configuration** box, paste the following configuration. + @@ -249,5 +249,5 @@ When you've finished experimenting with your data pipeline, you can delete the p * Choose xref:develop:connect/components/catalog.adoc[connectors for your use case]. * Learn how to xref:develop:connect/configuration/secret-management.adoc[add secrets to your pipeline]. * Learn how to xref:develop:connect/configuration/monitor-connect.adoc[monitor a data pipeline on a BYOC or Dedicated cluster]. -* Learn how to xref:develop:connect/configuration/scale-pipelines.adoc[manually scale resources for a pipeline on a BYOC or Dedicated cluster]. +* Learn how to xref:develop:connect/configuration/scale-pipelines.adoc[manually scale resources for a pipeline]. * Learn how to xref:redpanda-connect:guides:getting_started.adoc[configure, test, and run a data pipeline locally]. From 114b4e4470a669ec0f2702255444c99975700d2f Mon Sep 17 00:00:00 2001 From: Angela Simms Date: Thu, 20 Feb 2025 14:33:47 +0000 Subject: [PATCH 3/4] minor refinements --- .../pages/connect/configuration/resource-management.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/develop/pages/connect/configuration/resource-management.adoc b/modules/develop/pages/connect/configuration/resource-management.adoc index 3efbccb7..ae5bd8e3 100644 --- a/modules/develop/pages/connect/configuration/resource-management.adoc +++ b/modules/develop/pages/connect/configuration/resource-management.adoc @@ -6,14 +6,14 @@ == Prerequisites -- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated], xref:get-started:cluster-types/serverless-pro.adoc[Serverless Pro], or xref:get-started:cluster-types/serverless.adoc[Serverless Standard] cluster +- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated], xref:get-started:cluster-types/serverless-pro.adoc[Serverless Pro], or xref:get-started:cluster-types/serverless.adoc[Serverless Standard] cluster. - An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the xref:redpanda-connect:components:processors/benchmark.adoc[`benchmark` processor]. === Understanding compute units -A compute unit allocates a specific amount of server resource (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of up to 1 MB/sec. You can allocate up to a maximum of 18 compute units per pipeline. +A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of up to 1 MB/sec. -Note: Server resources are charged at an hourly rate in compute unit hours. +Server resources are charged at an hourly rate in compute unit hours and you can allocate up to a maximum of 18 compute units per pipeline. |=== | Number of compute units | CPU | Memory @@ -92,7 +92,7 @@ Note: Server resources are charged at an hourly rate in compute unit hours. |=== -NOTE: For pipelines with embedded Ollama AI components, one GPU is automatically allocated to the pipeline, which is equivalent to 30 compute units or 3.0 CPU (`3000m`) and 12 GB of memory (`12000M`). +NOTE: For pipelines with embedded Ollama AI components, one GPU is automatically allocated to the pipeline, which is equivalent to 30 compute units, or 3.0 CPU (`3000m`) and 12 GB of memory (`12000M`). === Set an initial resource limit From 0ea628daafa7ba5b70f0533f01f1f3929a1917f7 Mon Sep 17 00:00:00 2001 From: Angela Simms Date: Thu, 27 Feb 2025 17:22:04 +0000 Subject: [PATCH 4/4] Address review comments --- .../configuration/resource-management.adoc | 24 ++++++------------- .../pages/connect/connect-quickstart.adoc | 2 +- 2 files changed, 8 insertions(+), 18 deletions(-) diff --git a/modules/develop/pages/connect/configuration/resource-management.adoc b/modules/develop/pages/connect/configuration/resource-management.adoc index ae5bd8e3..5c63bd5a 100644 --- a/modules/develop/pages/connect/configuration/resource-management.adoc +++ b/modules/develop/pages/connect/configuration/resource-management.adoc @@ -6,14 +6,16 @@ == Prerequisites -- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated], xref:get-started:cluster-types/serverless-pro.adoc[Serverless Pro], or xref:get-started:cluster-types/serverless.adoc[Serverless Standard] cluster. +- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC), xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated], xref:get-started:cluster-types/serverless-pro.adoc[Serverless Pro] or xref:get-started:cluster-types/serverless.adoc[Serverless Standard] cluster. - An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the xref:redpanda-connect:components:processors/benchmark.adoc[`benchmark` processor]. === Understanding compute units -A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory, and provides a message throughput of up to 1 MB/sec. +A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory. -Server resources are charged at an hourly rate in compute unit hours and you can allocate up to a maximum of 18 compute units per pipeline. +For sizing purposes, one compute unit supports an estimated message throughput of 1 MB/sec. However, actual performance depends on the complexity of a pipeline, including the components it contains and processing it does. + +Server resources are charged at an hourly rate in compute unit hours, and you can allocate up to a maximum of 15 compute units per pipeline. |=== | Number of compute units | CPU | Memory @@ -78,18 +80,6 @@ Server resources are charged at an hourly rate in compute unit hours and you can | 1.5 CPU (`1500m`) | 6.0 GB (`6000M`) -| 16 -| 1.6 CPU (`1600m`) -| 6.4 GB (`6400M`) - -| 17 -| 1.7 CPU (`1700m`) -| 6.8 GB (`6800M`) - -| 18 -| 1.8 CPU (`1800m`) -| 7.2 GB (`7200M`) - |=== NOTE: For pipelines with embedded Ollama AI components, one GPU is automatically allocated to the pipeline, which is equivalent to 30 compute units, or 3.0 CPU (`3000m`) and 12 GB of memory (`12000M`). @@ -111,7 +101,7 @@ To set an initial resource limit: . Select the **Redpanda Connect** tab. . Click **Create pipeline**. . Enter details for your pipeline, including a short name and description. -. In the **Compute units** box, leave the default **1** compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate up to a maximum of 18 compute units. +. In the **Compute units** box, leave the default **1** compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate up to a maximum of 15 compute units. . Add your pipeline configuration and click **Create** to run it. === Scale resources @@ -155,7 +145,7 @@ Cloud UI:: . Log in to https://cloud.redpanda.com[Redpanda Cloud^]. . Go to the cluster where the pipeline is set up. . On the **Connect** page, select your pipeline and click **Edit**. -. In the **Compute units** box, update the number of compute units. One compute unit provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 18 compute units per pipeline. +. In the **Compute units** box, update the number of compute units. One compute unit provides a message throughput of approximately 1 MB/sec. For higher throughputs, you can allocate up to a maximum of 15 compute units per pipeline. . Click **Update** to apply your changes. The specified resources are available immediately. -- diff --git a/modules/develop/pages/connect/connect-quickstart.adoc b/modules/develop/pages/connect/connect-quickstart.adoc index 50168c22..79e93db3 100644 --- a/modules/develop/pages/connect/connect-quickstart.adoc +++ b/modules/develop/pages/connect/connect-quickstart.adoc @@ -95,7 +95,7 @@ All Redpanda Connect configurations use a YAML file split into three sections: . Go to the **Connect** page on your cluster and click **Create pipeline**. . In **Pipeline name**, enter **emailprocessor-pipeline** and add a short description. For example, **Transforms email data using a mutation processor**. -. In the **Compute units** box, leave the default value of **1**. Compute units are used to allocate server resources to a pipeline. One compute unit is equivalent to 0.1 CPU and 400 MB of memory, and provides a message throughput of approximately 1 MB/sec. +. In the **Compute units** box, leave the default value of **1**. Compute units are used to allocate server resources to a pipeline. One compute unit is equivalent to 0.1 CPU and 400 MB of memory. . In the **Configuration** box, paste the following configuration. +