Skip to content

Commit

Permalink
Add support for subscription scoped metric gathering
Browse files Browse the repository at this point in the history
  • Loading branch information
@kgeckhart
kgeckhart committed Dec 21, 2023
1 parent 63cabfb commit ccff42f
Show file tree
Hide file tree
Showing 7 changed files with 127 additions and 36 deletions.
2 changes: 2 additions & 0 deletions component/prometheus/exporter/azure/azure.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ type Arguments struct {
MetricHelpTemplate string `river:"metric_help_template,attr,optional"`
AzureCloudEnvironment string `river:"azure_cloud_environment,attr,optional"`
ValidateDimensions bool `river:"validate_dimensions,attr,optional"`
Regions []string `river:"regions,attr,optional"`
}

var DefaultArguments = Arguments{
Expand Down Expand Up @@ -78,5 +79,6 @@ func (a *Arguments) Convert() *azure_exporter.Config {
MetricHelpTemplate: a.MetricHelpTemplate,
AzureCloudEnvironment: a.AzureCloudEnvironment,
ValidateDimensions: a.ValidateDimensions,
Regions: a.Regions,
}
}
1 change: 1 addition & 0 deletions converter/internal/staticconvert/internal/build/azure_exporter.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,6 @@ func toAzureExporter(config *azure_exporter.Config) *azure.Arguments {
MetricHelpTemplate: config.MetricHelpTemplate,
AzureCloudEnvironment: config.AzureCloudEnvironment,
ValidateDimensions: config.ValidateDimensions,
Regions: config.Regions,
}
}
52 changes: 35 additions & 17 deletions docs/sources/flow/reference/components/prometheus.exporter.azure.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,28 @@ title: prometheus.exporter.azure

# prometheus.exporter.azure

The `prometheus.exporter.azure` component embeds [`azure-metrics-exporter`](https://github.com/webdevops/azure-metrics-exporter) to collect metrics from [Azure Monitor](https://azure.microsoft.com/en-us/products/monitor). The exporter uses [Azure Resource Graph](https://azure.microsoft.com/en-us/get-started/azure-portal/resource-graph/#overview) queries to identify resources for gathering metrics.
The `prometheus.exporter.azure` component embeds [`azure-metrics-exporter`](https://github.com/webdevops/azure-metrics-exporter) to collect metrics from [Azure Monitor](https://azure.microsoft.com/en-us/products/monitor).

The exporter supports all metrics defined by Azure Monitor. You can find the complete list of available metrics in the [Azure Monitor documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported).
Metrics for this integration are exposed with the template `azure_{type}_{metric}_{aggregation}_{unit}` by default. As an example,
the Egress metric for BlobService would be exported as `azure_microsoft_storage_storageaccounts_blobservices_egress_total_bytes`.

The exporter offers two options for gathering metrics,
1. (Default) Use an [Azure Resource Graph](https://azure.microsoft.com/en-us/get-started/azure-portal/resource-graph/#overview) query to identify resources for gathering metrics
1. This will make 1 API call per resource identified
1. Subscriptions with a reasonable amount of resources are liable to hit the [12000 requests per hour rate limit](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#subscription-and-tenant-limits) azure enforces
1. Set the regions to gather metrics from and get metrics for all resources across those regions
1. This will make an API call per subscription reducing the number of API calls dramatically
1. This approach does not work with all resource types and Azure's does not document which resource types do/do not work
1. A resource type which is not support will produce errors which look like `Resource type: microsoft.containerservice/managedclusters not enabled for Cross Resource metrics`

## Authentication

{{< param "PRODUCT_NAME" >}} must be running in an environment with access to Azure. The exporter uses the Azure SDK for go and supports [authentication](https://learn.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication?tabs=bash#2-authenticate-with-azure).

The account used by {{< param "PRODUCT_NAME" >}} needs:

- [Read access to the resources that will be queried by Resource Graph](https://learn.microsoft.com/en-us/azure/governance/resource-graph/overview#permissions-in-azure-resource-graph)
- When using an Azure Resoure Graph query, [read access to the resources that will be queried by Resource Graph](https://learn.microsoft.com/en-us/azure/governance/resource-graph/overview#permissions-in-azure-resource-graph)
- Permissions to call the [Microsoft.Insights Metrics API](https://learn.microsoft.com/en-us/rest/api/monitor/metrics/list) which should be the `Microsoft.Insights/Metrics/Read` permission

## Usage
Expand Down Expand Up @@ -51,23 +60,26 @@ prometheus.exporter.azure LABEL {
You can use the following arguments to configure the exporter's behavior.
Omitted fields take their default values.

| Name | Type | Description | Default | Required |
| ----------------------------- | -------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------- | -------- |
| `subscriptions` | `list(string)` | List of subscriptions to scrap metrics from. | | yes |
| `resource_type` | `string` | The Azure Resource Type to scrape metrics for. | | yes |
| `metrics` | `list(string)` | The metrics to scrape from resources. | | yes |
| `resource_graph_query_filter` | `string` | The [Kusto query][] filter to apply when searching for resources. | | no |
| `metric_aggregations` | `list(string)` | Aggregations to apply for the metrics produced. | | no |
| `timespan` | `string` | [ISO8601 Duration][] over which the metrics are being queried. | `"PT1M"` (1 minute) | no |
| `included_dimensions` | `list(string)` | List of dimensions to include on the final metrics. | | no |
| `included_resource_tags` | `list(string)` | List of resource tags to include on the final metrics. | `["owner"]` | no |
| `metric_namespace` | `string` | Namespace for `resource_type` which have multiple levels of metrics. | | no |
| `azure_cloud_environment` | `string` | Name of the cloud environment to connect to. | `"azurecloud"` | no |
| `metric_name_template` | `string` | Metric template used to expose the metrics. | `"azure_{type}_{metric}_{aggregation}_{unit}"` | no |
| `metric_help_template` | `string` | Description of the metric. | `"Azure metric {metric} for {type} with aggregation {aggregation} as {unit}"` | no |
| Name | Type | Description | Default | Required |
|-------------------------------|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|----------|
| `subscriptions` | `list(string)` | List of subscriptions to scrap metrics from. | | yes |
| `resource_type` | `string` | The Azure Resource Type to scrape metrics for. | | yes |
| `metrics` | `list(string)` | The metrics to scrape from resources. | | yes |
| `resource_graph_query_filter` | `string` | The [Kusto query][] filter to apply when searching for resources. Cannot be used if `regions` is set. | | no |
| `regions` | `list(string)` | The list of regions for gathering metrics and enables gathering metrics for all resources in the subscription. Cannot be used if `resource_graph_query_filter` is set. | | no |
| `metric_aggregations` | `list(string)` | Aggregations to apply for the metrics produced. | | no |
| `timespan` | `string` | [ISO8601 Duration][] over which the metrics are being queried. | `"PT1M"` (1 minute) | no |
| `included_dimensions` | `list(string)` | List of dimensions to include on the final metrics. | | no |
| `included_resource_tags` | `list(string)` | List of resource tags to include on the final metrics. | `["owner"]` | no |
| `metric_namespace` | `string` | Namespace for `resource_type` which have multiple levels of metrics. | | no |
| `azure_cloud_environment` | `string` | Name of the cloud environment to connect to. | `"azurecloud"` | no |
| `metric_name_template` | `string` | Metric template used to expose the metrics. | `"azure_{type}_{metric}_{aggregation}_{unit}"` | no |
| `metric_help_template` | `string` | Description of the metric. | `"Azure metric {metric} for {type} with aggregation {aggregation} as {unit}"` | no |

The list of available `resource_type` values and their corresponding `metrics` can be found in [Azure Monitor essentials][].

The list of available `regions` to your subscription can be found by running the azure CLI command `az account list-locations --query '[].name'`.

The `resource_graph_query_filter` can be embedded into a template query of the form `Resources | where type =~ "<resource_type>" <resource_graph_query_filter> | project id, tags`.

Valid values for `metric_aggregations` are `minimum`, `maximum`, `average`, `total`, and `count`. If no aggregation is specified, the value is retrieved from the metric. For example, the aggregation value of the metric `Availability` in [Microsoft.ClassicStorage/storageAccounts](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/microsoft-classicstorage-storageaccounts-metrics) is `average`.
Expand Down Expand Up @@ -107,6 +119,9 @@ debug metrics.
prometheus.exporter.azure "example" {
subscriptions = SUBSCRIPTIONS
resource_type = "Microsoft.Storage/storageAccounts"
regions = [
"westeurope",
]
metric_namespace = "Microsoft.Storage/storageAccounts/blobServices"
metrics = [
"Availability",
Expand All @@ -120,8 +135,11 @@ prometheus.exporter.azure "example" {
"SuccessServerLatency",
"Transactions",
]
included_dimensions = [
"ApiName",
"TransactionType",
]
timespan = "PT1H"
resource_graph_query_filter = "where location == 'westeurope'"
}
// Configure a prometheus.scrape component to send metrics to.
Expand Down
34 changes: 29 additions & 5 deletions docs/sources/static/configuration/integrations/azure-exporter-config.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,16 @@ title: azure_exporter_config
## Overview
The `azure_exporter_config` block configures the `azure_exporter` integration, an embedded version of
[`azure-metrics-exporter`](https://github.com/webdevops/azure-metrics-exporter), used to
collect metrics from [Azure Monitor](https://azure.microsoft.com/en-us/products/monitor). The
exporter uses [Azure Resource Graph](https://azure.microsoft.com/en-us/get-started/azure-portal/resource-graph/#overview)
queries to identify resources for gathering metrics.
collect metrics from [Azure Monitor](https://azure.microsoft.com/en-us/products/monitor).

The exporter offers two options for gathering metrics,
1. (Default) Use an [Azure Resource Graph](https://azure.microsoft.com/en-us/get-started/azure-portal/resource-graph/#overview) query to identify resources for gathering metrics
1. This will make 1 API call per resource identified
1. Subscriptions with a reasonable amount of resources are liable to hit the [12000 requests per hour rate limit](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#subscription-and-tenant-limits) azure enforces
1. Set the regions to gather metrics from and get metrics for all resources across those regions
1. This will make an API call per subscription reducing the number of API calls dramatically
1. This approach does not work with all resource types and Azure's does not document which resource types do/do not work
1. A resource type which is not support will produce errors which look like `Resource type: microsoft.containerservice/managedclusters not enabled for Cross Resource metrics`

## List of Supported Services and Metrics
The exporter supports all metrics defined by Azure Monitor. The complete list of available metrics can be found in the [Azure Monitor documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported).
Expand Down Expand Up @@ -95,7 +102,14 @@ The account used by Grafana Agent needs:

# Optional: The [kusto query](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/) filter to apply when searching for resources
# This value will be embedded in to a template query of the form `Resources | where type =~ "<resource_type>" <resource_graph_query_filter> | project id, tags`
# Cannot be used if `regions` is set.
[resource_graph_query_filter: <string>]

# Optional: The list of regions for gathering metrics. Enables gather metrics for all resources in the subscription.
# The list of available `regions` to your subscription can be found by running the azure CLI command `az account list-locations --query '[].name'`.
# Cannot be used if `resource_graph_query_filter` is set.
regions:
[ - <string> ... ]

# Optional: Aggregation to apply for the metrics produced. Valid values are minimum, maximum, average, total, and count
# If no aggregation is specified the value for `Aggregation Type` on the `Metric` is used from https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported
Expand Down Expand Up @@ -167,6 +181,7 @@ The account used by Grafana Agent needs:
included_dimensions:
- node
- nodepool
- device
```
#### Blob Storage Metrics
Expand All @@ -178,6 +193,8 @@ The account used by Grafana Agent needs:
- <subscription_id>
resource_type: Microsoft.Storage/storageAccounts
metric_namespace: Microsoft.Storage/storageAccounts/blobServices
regions:
- westeurope
metrics:
- Availability
- BlobCapacity
Expand All @@ -189,8 +206,10 @@ The account used by Grafana Agent needs:
- SuccessE2ELatency
- SuccessServerLatency
- Transactions
included_dimensions:
- ApiName
- TransactionType
timespan: PT1H
resource_graph_query_filter: where location == "westeurope"
```
### Multiple Azure Services in a single config
Expand Down Expand Up @@ -222,6 +241,8 @@ metrics:
subscriptions:
- 179c4f30-ebd8-489e-92bc-fb64588dadb3
resource_type: ["Microsoft.Storage/storageAccounts"]
regions:
- westeurope
metric_namespace: ["Microsoft.Storage/storageAccounts/blobServices"]
metrics:
- Availability
Expand All @@ -234,8 +255,10 @@ metrics:
- SuccessE2ELatency
- SuccessServerLatency
- Transactions
included_dimensions:
- ApiName
- TransactionType
timespan: ["PT1H"]
resource_graph_query_filter: ["where location == 'westeurope'"]
- job_name: azure-kubernetes-node
scrape_interval: 1m
scrape_timeout: 50s
Expand Down Expand Up @@ -263,6 +286,7 @@ metrics:
included_dimensions:
- node
- nodepool
- device
```

In this example, all `azure_exporter`-specific configuration settings have been moved to the `scrape_config`. This method supports all available configuration options except `azure_cloud_environment`, which must be configured on the `azure_exporter`. For this method, if a field supports a singular value like `resource_graph_query_filter`, you
Expand Down
Loading

0 comments on commit ccff42f

Please sign in to comment.