Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Update local-block processor config for 2.6 changes #4223

Merged
merged 1 commit into from
Oct 22, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 33 additions & 14 deletions docs/sources/tempo/operations/traceql-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,14 @@ To use the metrics generated from traces, you need to:

* Set the `local-blocks` processor to active in your `metrics-generator` configuration
* Configure a Tempo data source in Grafana or Grafana Cloud
* Access Grafana Cloud or Grafana version 10.4 or newer
* Access Grafana Cloud or Grafana version 10.4 or later

## Activate and configure the `local-blocks` processor

The local-blocks processor must be enabled to start using metrics queries like `{ } | rate()`.
If not enabled, then the metrics queries fail with the error `localblocks processor not found`.
Enabling the `local-blocks` processor can be done either per tenant or in all tenants.

To activate the `local-blocks` processor for all users, add it to the list of processors in the `overrides` block of your Tempo configuration.

```yaml
Expand All @@ -38,30 +42,45 @@ overrides:
metrics_generator_processors: ['local-blocks']
```

To configure the processor per tenant, use the `metrics_generator.processor` override.
To configure the processor per tenant, use the `metrics_generator_processor` override.

Example for per-tenant in the per-tenant overrides:

For more information about overrides, refer to [Standard overrides]({{< relref "../configuration#standard-overrides" >}}).
```yaml
overrides:
'tenantID':
metrics_generator_processors:
- local-blocks
```

### Configure the processor
By default, for all tenants in the main configuration:

Next, configure the `local-blocks` processor to record all spans for TraceQL metrics.
Here is an example configuration:
```yaml
overrides:
defaults:
metrics_generator:
processors: [local-blocks]
```

Add this configuration to run TraceQL metrics queries against all spans (and not just server spans):

```yaml
metrics_generator:
metrics_generator:
processor:
local_blocks:
filter_server_spans: false
storage:
path: /var/tempo/generator/wal
traces_storage:
path: /var/tempo/generator/traces
```

If you configured Tempo using the `tempo-distributed` Helm chart, you can also set `traces_storage` using your `values.yaml` file. Refer to the [Helm chart for an example](https://github.com/grafana/helm-charts/blob/559ecf4a9c9eefac4521454e7a8066778e4eeff7/charts/tempo-distributed/values.yaml#L362).
To run metrics queries on historical data, you must configure the local-blocks processor to flush RF1 blocks to object storage:

```yaml
metrics_generator:
processor:
local_blocks:
flush_to_storage: true
```

Refer to the [metrics-generator configuration]({{< relref "../configuration#metrics-generator" >}}) documentation for more information.
For more information about overrides, refer to [Standard overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#standard-overrides).

## Evaluate query timeouts

Expand Down Expand Up @@ -109,4 +128,4 @@ query_frontend:
metrics:
concurrent_jobs: 8
target_bytes_per_job: 1.25e+09 # ~1.25GB
```
```