-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[receiver/huaweicloudces]: Add huaweicloudces receiver (#36258)
#### Description Introduce a new receiver, huaweicloudces receiver. This is the first PR as skeleton for the receiver. #### Link to tracking issue Related to #34953 #### Testing N/A <!--Describe the documentation added.--> #### Documentation metadata.yaml, README.
- Loading branch information
Showing
18 changed files
with
699 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Use this changelog template to create an entry for release notes. | ||
|
|
||
| # One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' | ||
| change_type: new_component | ||
|
|
||
| # The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver) | ||
| component: huaweicloudcesreceiver | ||
|
|
||
| # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). | ||
| note: Introduce new receiver fetching data from huawei Cloud Eye Service. | ||
|
|
||
| # Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists. | ||
| issues: [34953] | ||
|
|
||
| # (Optional) One or more lines of additional information to render under the primary note. | ||
| # These lines will be padded with 2 spaces and then inserted directly into the document. | ||
| # Use pipe (|) for multiline entries. | ||
| subtext: | ||
|
|
||
| # If your change doesn't affect end users or the exported elements of any package, | ||
| # you should instead start your pull request title with [chore] or use the "Skip Changelog" label. | ||
| # Optional: The change log or logs in which this entry should be included. | ||
| # e.g. '[user]' or '[user, api]' | ||
| # Include 'user' if the change is relevant to end users. | ||
| # Include 'api' if there is a change to a library API. | ||
| # Default: '[user]' | ||
| change_logs: [] |
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| include ../../Makefile.Common |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,267 @@ | ||
| # Huawei Cloud CES Receiver | ||
|
|
||
| <!-- status autogenerated section --> | ||
| | Status | | | ||
| | ------------- |-----------| | ||
| | Stability | [development]: metrics | | ||
| | Distributions | [] | | ||
| | Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fhuaweicloudces) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fhuaweicloudces) | | ||
| | [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@heitorganzeli](https://www.github.com/heitorganzeli), [@narcis96](https://www.github.com/narcis96), [@mwear](https://www.github.com/mwear) | | ||
|
|
||
| [development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development | ||
| <!-- end autogenerated section --> | ||
|
|
||
| This receiver contains the implementation of the Huawei Cloud [Cloud Eye Service](https://www.huaweicloud.com/intl/en-us/product/ces.html) (CES) receiver for the OpenTelemetry Collector. The receiver collects metrics from Huawei Cloud's CES service and sends them to the OpenTelemetry Collector for processing and exporting. | ||
|
|
||
| ## Configuration | ||
|
|
||
| The following settings are required: | ||
|
|
||
| - `region_id`: The ID of the Huawei Cloud region from which metrics are collected. For example, `eu-west-101`. The full list of the available regions can be found [here](https://pkg.go.dev/github.com/huaweicloud/[email protected]/services/ces/v1/region). | ||
|
|
||
| - `project_id`: The ID of the project in Huawei Cloud. This is used to identify which project's metrics are to be collected. See [Obtaining a Project ID](https://support.huaweicloud.com/intl/en-us/devg-apisign/api-sign-provide-proid.html). | ||
|
|
||
| - `period`: The aggregation granularity of metrics retrieved from CES in seconds. For details about the aggregation, see [What Is Rollup?](https://support.huaweicloud.com/intl/en-us/ces_faq/ces_faq_0009.html). Possible values are 1, 300, 1200, 3600, 14400, and 86400. | ||
| - 1: Cloud Eye performs no aggregation and displays raw data. | ||
| - 300: Cloud Eye aggregates data every 5 minutes. | ||
| - 1200: Cloud Eye aggregates data every 20 minutes. | ||
| - 3600: Cloud Eye aggregates data every hour. | ||
| - 14400: Cloud Eye aggregates data every 4 hours. | ||
| - 86400: Cloud Eye aggregates data every 24 hours. | ||
| - `filter`: The filter field determines the aggregation method used for the metrics. This aggregation is applied to the data points within the specified period. Valid values for filter include: | ||
|
|
||
| - `average`: Calculates the average value over the period. | ||
| - `min`: Retrieves the minimum value within the period. | ||
| - `max`: Retrieves the maximum value within the period. | ||
| - `sum`: Computes the sum of all data points within the period. | ||
| - `variance`: Calculates the variance (square of the standard deviation) for the data points. | ||
|
|
||
| - `no_verify_ssl`: A boolean flag indicating whether SSL verification should be disabled. Set to True to disable SSL verification. | ||
|
|
||
| - `access_key`: The access key needed for CES authentification. Check `Huawei Cloud SDK Authentication Setup` section for more details. | ||
|
|
||
| - `secret_key`: The secret key needed for CES authentification. Check `Huawei Cloud SDK Authentication Setup` section for more details. | ||
|
|
||
| The following settings are optional: | ||
|
|
||
| - `initial_delay`: The delay before the first collection of metrics begins. This is a duration field, such as 5s for 5 seconds. | ||
|
|
||
| - `collection_interval` (default = `60s`): This is the interval at which this receiver collects metrics. This value must be a string readable by Golang's [time.ParseDuration](https://pkg.go.dev/time#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. We recommend a polling interval of at least one minute. | ||
|
|
||
| - `retry_on_failure`: The following configurations can be used to control the retry policy of the CES client. The default values are suitable for most deployment scenarios. | ||
| - `enabled` (default true) | ||
| - `initial_interval` (default 100ms) | ||
| - `max_interval` (default 1s) | ||
| - `max_elapsed_time` (default 15s) | ||
| - `randomization_factor` (default 0.5) | ||
| - `multiplier` (default 1.5) | ||
|
|
||
| ### Example Configuration | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| huaweicloudcesreceiver: | ||
| collection_interval: 3h | ||
| initial_delay: 5s | ||
| region_id: eu-west-101 | ||
| access_key: ${env:HUAWEICLOUD_SDK_AK} | ||
| secret_key: ${env:HUAWEICLOUD_SDK_SK} | ||
| project_id: "project_1" | ||
| period: 300 | ||
| filter: average | ||
| no_verify_ssl: True | ||
| ``` | ||
| The full list of settings exposed for this receiver are documented [here](./config.go). | ||
| ### Huawei Cloud SDK Authentication Setup | ||
| To ensure secure authentication, the Access Key (AK) and Secret Key (SK) used by the Huawei Cloud SDK must be stored in environment variables. See [Obtaining an AK/SK](https://support.huaweicloud.com/intl/en-us/devg-apisign/api-sign-provide-aksk.html). | ||
| Before running the application, you need to set the environment variables `HUAWEICLOUD_SDK_AK` and `HUAWEICLOUD_SDK_SK` in your local environment. Here’s how you can do it: | ||
|
|
||
| 1. Open your terminal. | ||
| 2. Set the environment variables by executing the following commands: | ||
|
|
||
| ```sh | ||
| export HUAWEICLOUD_SDK_AK=your-access-key | ||
| export HUAWEICLOUD_SDK_SK=your-secret-key | ||
| ``` | ||
|
|
||
| 3. Verify that the variables are set correctly: | ||
|
|
||
| ```sh | ||
| echo $HUAWEICLOUD_SDK_AK | ||
| echo $HUAWEICLOUD_SDK_SK | ||
| ``` | ||
|
|
||
| ## Error handling | ||
| If you encounter any errors, please refer to: | ||
| - [Huawei Cloud Error Codes](https://support.huaweicloud.com/intl/en-us/devg-apisign/api-sign-errorcode.html) | ||
| - [CES Error Codes](https://support.huaweicloud.com/intl/en-us/api-ces/ErrorCode.html) | ||
| - [Quota management](https://support.huaweicloud.com/intl/en-us/usermanual-ces/en-us_topic_0154940152.html) | ||
|
|
||
|
|
||
| ## Converting CES metric representation to Open Telementery metric representation | ||
|
|
||
|
|
||
| | Source Field | Target Field | Description | | ||
| |--------------------------|----------------------------------------------|-------------------------------------------------------------------------------------------------------| | ||
| | **metric_name** | `scoped_metric.metric.name` | The name of the metric. | | ||
| | **unit** | `scoped_metric.metric.unit` | The unit of the metric, e.g., `%`, `bit/s`. | | ||
| | **datapoints.value** | `scoped_metric.metric.gauge.data_points.value` | The value of the metric at the specified timestamp. | | ||
| | **datapoints.timestamp** | `scoped_metric.metric.gauge.data_points.timestamp` | The timestamp of the metric in nanoseconds. Converted from milliseconds by multiplying by 1,000,000. | | ||
| | **dimensions.name** | `key of scoped_metric.metric.metadata` | The dimension name stored as an attribute. | | ||
| | **dimensions.value** | `value of scoped_metric.metric.metadata` | The dimension value stored as an attribute. | | ||
| | **Receiver Config** | `resource.attributes["project.id"]` | The project ID used in the configuration file of the receiver. | | ||
| | **Receiver Config** | `resource.attributes["region.id"]` | The region ID used in the configuration file of the receiver. | | ||
| | **namespace** | `resource.attributes["service.namespace"]` | The namespace of the metric, e.g., `SYS.VPC`, stored as an attribute with the key `service.namespace`. | | ||
| | *N/A* | `resource.attributes["cloud.provider"]` | Set to `"huawei_cloud"` as the cloud provider. | | ||
| | *N/A* | `scoped_metric.scope.name` | Set to `"huawei_cloud_ces"` as the scope name. | | ||
| | *N/A* | `scoped_metric.scope.version` | Set to `"v1"` as the scope version. | | ||
| | | ||
|
|
||
| ### Notes | ||
|
|
||
| - The `timestamp` field in the source is converted from milliseconds to nanoseconds in the target field. | ||
| - Some fields are added in the target format with constant values to provide additional context and metadata. | ||
|
|
||
| ### Example: | ||
|
|
||
| ```json | ||
| [ | ||
| { | ||
| "unit": "%", | ||
| "datapoints": [ | ||
| { "average": 10, "timestamp": 1722580500000 }, | ||
| { "average": 20, "timestamp": 1722580800000 } | ||
| ], | ||
| "namespace": "SYS.ECS", | ||
| "metric_name": "cpu_util", | ||
| "dimensions": [ | ||
| { | ||
| "name": "instance_id", | ||
| "value": "faea5b75-e390-4e2b-8733-9226a9026070" | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "unit": "%", | ||
| "datapoints": [ | ||
| { "average": 30, "timestamp": 1722580500000 }, | ||
| { "average": 40, "timestamp": 1722580800000 } | ||
| ], | ||
| "namespace": "SYS.ECS", | ||
| "metric_name": "mem_util", | ||
| "dimensions": [ | ||
| { | ||
| "name": "instance_id", | ||
| "value": "abcea5b75-e390-4e2b-8733-9226a9026070" | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "unit": "bit/s", | ||
| "datapoints": [ | ||
| { "average": 1024, "timestamp": 1722580500000}, | ||
| { "average": 2048, "timestamp": 1722580800000} | ||
| ], | ||
| "namespace": "SYS.VPC", | ||
| "metric_name": "upstream_bandwidth_usage", | ||
| "dimensions": [ | ||
| { | ||
| "name": "publicip_id", | ||
| "value": "test-baae-4dd9-ad3f-1234" | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| ``` | ||
|
|
||
| converts to | ||
|
|
||
| ```json | ||
| { | ||
| "resourceMetrics": [ | ||
| { | ||
| "resource": { | ||
| "attributes": { | ||
| "cloud.provider": "huawei_cloud", | ||
| "project.id": "project_1", | ||
| "region.id": "eu-west-101", | ||
| "system.namespace": "SYS.ECS", | ||
| } | ||
| }, | ||
| "scopeMetrics": [ | ||
| { | ||
| "scope": { "name": "huawei_cloud_ces", "version": "v1" }, | ||
| "metrics": [ | ||
| { | ||
| "name": "cpu_util", | ||
| "unit": "%", | ||
| "gauge": { | ||
| "dataPoints": [ | ||
| { "timeUnixNano": "1722580500000000000", "asDouble": 10 }, | ||
| { "timeUnixNano": "1722580800000000000", "asDouble": 20 } | ||
| ] | ||
| }, | ||
| "metadata": { | ||
| "instance_id": "faea5b75-e390-4e2b-8733-9226a9026070", | ||
| } | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "scope": { "name": "huawei_cloud_ces", "version": "v1" }, | ||
| "metrics": [ | ||
| { | ||
| "name": "mem_util", | ||
| "unit": "%", | ||
| "gauge": { | ||
| "dataPoints": [ | ||
| { "timeUnixNano": "1722580500000000000", "asDouble": 30 }, | ||
| { "timeUnixNano": "1722580800000000000", "asDouble": 40 } | ||
| ] | ||
| }, | ||
| "metadata": { | ||
| "instance_id": "abcea5b75-e390-4e2b-8733-9226a9026070", | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "resource": { | ||
| "attributes": { | ||
| "cloud.provider": "huawei_cloud", | ||
| "project.id": "project_1", | ||
| "region.id": "eu-west-101", | ||
| "system.namespace": "SYS.VPC", | ||
| } | ||
| }, | ||
| "scopeMetrics": [ | ||
| { | ||
| "scope": { "name": "huawei_cloud_ces", "version": "v1" }, | ||
| "metrics": [ | ||
| { | ||
| "name": "upstream_bandwidth_usage", | ||
| "unit": "bits/s", | ||
| "gauge": { | ||
| "dataPoints": [ | ||
| { "timeUnixNano": "1722580500000000000", "asDouble": 1024 }, | ||
| { "timeUnixNano": "1722580800000000000", "asDouble": 2048 } | ||
| ] | ||
| }, | ||
| "metadata": { | ||
| "publicip_id": "test-baae-4dd9-ad3f-1234", | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ``` |
Oops, something went wrong.