-
Notifications
You must be signed in to change notification settings - Fork 487
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a "When to use Flow" section to the docs #5793
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
--- | ||
description: Learn how to Flow compares to other offerings | ||
menuTitle: When to use Flow | ||
title: When to use Flow | ||
weight: 25 | ||
--- | ||
|
||
# When to use Flow | ||
|
||
Grafana Agent Flow allows you to process telemetry in a variety of formats, natively: | ||
* [OTLP][] (`otelcol` components) | ||
* Prometheus (`prometheus` components) | ||
* Loki (`loki` components) | ||
* Pyroscope (`pyroscope` components) | ||
|
||
Native processing of telemetry could lead to higher efficiency and ease of use. | ||
|
||
It is also possible to switch from one format to another. For example: | ||
* `otelcol.exporter.prometheus` converts OTLP metrics to Prometheus logs. | ||
* `otelcol.receiver.prometheus` converts Prometheus metrics to OTLP metrics. | ||
* `otelcol.exporter.loki` converts OTLP logs to Loki logs. | ||
* `otelcol.receiver.loki` converts Loki logs to OTLP logs. | ||
* `otelcol.connector.spanlogs` converts OTLP spans to OTLP logs. | ||
* `otelcol.connector.spanmetrics` converts OTLP spans to OTLP metrics. | ||
|
||
There are even Flow components that do not deal with telemetry. For example, `mimir.rules.kubernetes` | ||
can be used to configure a Mimir instance. | ||
|
||
The following topics describe in more detail how Flow compares to similar products: | ||
|
||
{{< section >}} | ||
|
||
[OTLP]: https://grafana.com/docs/grafana-cloud/send-data/otlp/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
--- | ||
description: Learn how to Flow compares to the OpenTelemetry Collector | ||
menuTitle: OpenTelemetry Collector | ||
title: OpenTelemetry Collector | ||
weight: 100 | ||
--- | ||
|
||
# Comparing Flow with the OpenTelemetry Collector | ||
|
||
## Which one of the two should you use? | ||
|
||
Flow may be a better choice if: | ||
* You prefer River syntax over YAML. | ||
* You would like to collect "profile" telemetry - Flow can do this via Pyroscope components. | ||
* You would like to process Prometheus or Loki telemetry without converting it to OTLP. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this the OSS documentation? One extra advantage of Flow is that Grafana Labs offers support for it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is for OSS, but sometimes the Cloud docs import OSS docs and I'm not sure if this one will end up being included. |
||
|
||
The Collector may be a better choice if: | ||
* You are already using the Collector, and do not need any of the features of Flow. | ||
* You prefer YAML syntax over River. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is the main point the fact that you can add a custom component? Or is using the Collector Builder the advantage? Either way, there is nothing stopping someone from forking the Agent and adding their own custom component. We just don't document that process very well, because in the long term we want to solve by developing a plugin-based system which doesn't require a new distribution/fork when a custom component is needed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The use case is having a custom component, having a supported (and documented?) way to do that is what determines the tooling. |
||
|
||
## Similarities | ||
|
||
### Many Collector components are available in Flow | ||
|
||
Flow contains many of the components available inside the Collector. | ||
For example, Collector's OTLP exporter is available in Flow as `otelcol.exporter.otlp`. | ||
|
||
The Grafana Agent development team is actively working on adding more of the Collector's components to Flow. | ||
If a Flow user needs a Collector feature to be made available in the Agent, the Agent development team could work on implementing it. | ||
|
||
### Similar performance when processing OpenTelemetry signals natively | ||
|
||
Most of Flow's `otelcol` components are just thin wrappers over a Collector component. | ||
As a result, the CPU and memory performance of Flow is similar. | ||
|
||
## Differences | ||
|
||
### Configuration language | ||
|
||
Collector uses YAML for its configuration, and Flow uses River for its configuration. | ||
|
||
#### Example - coalesce | ||
|
||
One of River's main advantages is its standard library. It contains handy functions such as coalesce. | ||
You can use this function to retrieve the first argument which is not null or empty: | ||
|
||
```river | ||
``` | ||
|
||
### Modularity | ||
|
||
The Agent configuration is flexible and modular: | ||
* Telemetry and non-telemetry components can be combined easily. | ||
For example, a configuration argument for `otelcol.exporter.otlp` could be supplied from the response of an HTTP request via `remote.http`. | ||
* Some or all of the components which an Agent runs could be defined outside of its main config file. | ||
For example, whole snippets of Flow configuration could be loaded from a file (`module.file`), | ||
a Git repository (`module.git`), or a HTTP endpoint (`module.http`). | ||
|
||
#### Example - retrieving data from a file | ||
|
||
Let's say you want to use OAuth2 authentication in the Collector. If you need to retrieve `client_id` | ||
or `client_secret` from a file, then you must use the `client_id_file` or `client_secret_file` config parameters. | ||
|
||
```yaml | ||
``` | ||
|
||
In the Agent, you'd use `otelcol.auth.oauth2` with the normal `client_id` and `client_secret` parameters, | ||
and you would set up another component that retrieves those from a `local.file` component. | ||
|
||
```river | ||
``` | ||
|
||
The string could also come from a `remote.s3`, `remote.vault`, or `remote.http`. | ||
This gives Flow users lots of flexibility because strings coming out of those components | ||
could be used for any parameter in any component which requires a string - not just for | ||
a `client_id` for OAuth2. | ||
|
||
```river | ||
``` | ||
|
||
### Flow can process Prometheus signals natively | ||
|
||
Collector needs to convert Prometheus signals to the OTLP format to process them. | ||
Flow, on the other hand, can process those signals natively using components such as `prometheus.relabel`, `prometheus.relabel`, and `prometheus.remote_write`. | ||
This could lead to better performance and ease of use. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Our tests have shown that this leads to considerable performance gains. @sh0rez , is there anything public we can link to? |
||
|
||
### Flow documentation is consistent and structured | ||
|
||
Flow components are documented in a consistent and structured way. | ||
|
||
### Some Collector features are not available in the Agent and vice-versa | ||
|
||
Agent doesn't have all the components which the Collector has. However, the Grafana Agent development team is constantly working on | ||
adding new components, and would be happy to add new components that Flow users need. | ||
|
||
### Flow supports "profile" telemetry signals | ||
|
||
OpenTelemetry currently does not support "profile" signals. Flow supports "profile" signals through components such as `pyroscope.scrape` and `pyroscope.ebpf`. | ||
|
||
### Flow can load balance scraping dynamically | ||
|
||
Using clustering, `prometheus.scrape` and `pyroscope.scrape` use a gossip communication to load balance | ||
scraping among each other. Agents can be added and removed from the cluster without having to restart the whole set of Agents. | ||
|
||
### Flow is usually a few versions of OpenTelemetry behind | ||
|
||
Flow imports OpenTelemetry libraries as a dependency. Generally, a given version of Flow might use OpenTelemetry libraries | ||
which are older by 1, 2, or even 4 months. This is usually not a problem and the Agent development team could schedule an | ||
upgrade of the Agent's OpenTelemetry dependencies outside of the usual upgrade cycle if it is important to a user. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
--- | ||
description: Learn how to Flow compares to the Prometheus Agent Mode | ||
menuTitle: Prometheus Agent Mode | ||
title: Prometheus Agent Mode | ||
weight: 200 | ||
--- | ||
|
||
## Comparing Flow with the Prometheus Agent Mode | ||
|
||
Grafana Agent Flow supports all of the [features][prom-agent-mode-blog] of the [Prometheus Agent Mode][prom-agent-mode-flag] via components such as: | ||
* `discovery.http` | ||
* `discovery.relabel` | ||
* `prometheus.scrape` | ||
* `prometheus.relabel` | ||
* `prometheus.remote_write` | ||
|
||
Agent Flow's performance is comparable to Prometheus, because `prometheus` Flow components are able | ||
to processes Prometheus metrics natively without converting them to other formats such as OTLP. | ||
|
||
Agent Flow also comes pre-built with exporters out of the box. For example, `prometheus.exporter.unix` | ||
provides the functionality of Prometheus' [Node Exporter][node-exporter]. | ||
|
||
[prom-agent-mode-flag]: https://prometheus.io/docs/prometheus/latest/feature_flags/#prometheus-agent | ||
[prom-agent-mode-blog]: https://prometheus.io/blog/2021/11/16/agent/ | ||
[node-exporter]: https://github.com/prometheus/node_exporter |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to add a Venn diagram such as this one in the beginning of this page: