Add first chapter of flow-by-example to docs (#5888)

* Add first chapter of flow-by-example to docs * Update strucutre to leaf bundles, move some assets to shared directory, add docker-compose and update FAQ * Remove unnecessary assets and change to /media/ links * Apply suggestions from code review Co-authored-by: Clayton Cornell <[email protected]> * Add note admonitions and fix some links * Rename faq.md -> get-started.md and update description * Replace 'Grafana Agent in Flow mode' with PRODUCT_NAME param * Remove solution file and use a collapse shortcode with a code block instead * Change to repetitive list and shift paragraph content to new line for each * Switch Headline Case -> Sentence case * Replace relrefs with full urls * Add <AGENT_VERSION> to links * Remove docker-compose.yaml and download and add it as a code block to get-started * Add weights to fix ordering --------- Co-authored-by: Clayton Cornell <[email protected]>
grafana · Jan 15, 2024 · 77aac50 · 77aac50
1 parent c054f30
commit 77aac50
Show file tree

Hide file tree

Showing 3 changed files with 380 additions and 0 deletions.
diff --git a/docs/sources/flow/tutorials/flow-by-example/_index.md b/docs/sources/flow/tutorials/flow-by-example/_index.md
@@ -0,0 +1,17 @@
+---
+aliases:
+- /docs/grafana-cloud/agent/flow/tutorials/flow-by-example/
+- /docs/grafana-cloud/monitor-infrastructure/agent/flow/tutorials/flow-by-example/
+- /docs/grafana-cloud/monitor-infrastructure/integrations/agent/flow/tutorials/flow-by-example/
+- /docs/grafana-cloud/send-data/agent/flow/tutorials/flow-by-example/
+canonical: https://grafana.com/docs/agent/latest/flow/tutorials/flow-by-example/
+description: Learn how to use Grafana Agent Flow
+title: Flow by example
+weight: 300
+---
+
+# Flow by example
+
+This section provides a set of step-by-step tutorials that show how to use {{< param "PRODUCT_NAME" >}}.
+
+{{< section >}}
diff --git a/docs/sources/flow/tutorials/flow-by-example/first-components-and-stdlib/index.md b/docs/sources/flow/tutorials/flow-by-example/first-components-and-stdlib/index.md
@@ -0,0 +1,274 @@
+---
+aliases:
+- /docs/grafana-cloud/agent/flow/tutorials/flow-by-example/first-components-and-stdlib/
+- /docs/grafana-cloud/monitor-infrastructure/agent/flow/tutorials/flow-by-example/first-components-and-stdlib/
+- /docs/grafana-cloud/monitor-infrastructure/integrations/agent/flow/tutorials/flow-by-example/first-components-and-stdlib/
+- /docs/grafana-cloud/send-data/agent/flow/tutorials/first-components-and-stdlib/
+canonical: https://grafana.com/docs/agent/latest/flow/tutorials/flow-by-example/first-components-and-stdlib/
+description: Learn about the basics of River and the Flow configuration language
+title: First components and introducing the standard library
+weight: 20
+---
+
+# First components and the standard library
+
+This tutorial covers the basics of the River language and the standard library. It introduces a basic pipeline that collects metrics from the host and sends them to Prometheus.
+
+## River basics
+
+[Configuration language]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/config-language/
+[Configuration language concepts]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/concepts/configuration_language/
+[Standard library documentation]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/reference/stdlib/
+
+**Recommended reading**
+
+- [Configuration language][]
+- [Configuration language concepts][]
+
+[River](https://github.com/grafana/river) is an HCL-inspired configuration language used to configure {{< param "PRODUCT_NAME" >}}. A River file is comprised of three things:
+
+1. **Attributes**
+
+   `key = value` pairs used to configure individual settings.
+
+    ```river
+    url = "http://localhost:9090"
+    ```
+
+1. **Expressions**
+
+   Expressions are used to compute values. They can be constant values (for example, `"localhost:9090"`), or they can be more complex (for example, referencing a component's export: `prometheus.exporter.unix.targets`. They can also be a mathematical expression: `(1 + 2) * 3`, or a standard library function call: `env("HOME")`). We will use more expressions as we go along the examples. If you are curious, you can find a list of available standard library functions in the [Standard library documentation][].
+
+1. **Blocks**
+
+   Blocks are used to configure components with groups of attributes or nested blocks. The following example block can be used to configure the logging output of {{< param "PRODUCT_NAME" >}}:
+
+    ```river
+    logging {
+        level  = "debug"
+        format = "json"
+    }
+    ```
+
+    {{% admonition type="note" %}}
+The default log level is `info` and the default log format is `logfmt`.
+    {{% /admonition %}}
+
+    Try pasting this into `config.river` and running `/path/to/agent run config.river` to see what happens.
+
+    Congratulations, you've just written your first River file! You've also just written your first {{< param "PRODUCT_NAME" >}} configuration file. This configuration won't do anything, so let's add some components to it.
+
+    {{% admonition type="note" %}}
+Comments in River are prefixed with `//` and are single-line only. For example: `// This is a comment`.
+    {{% /admonition %}}
+
+## Components
+
+[Components]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/concepts/components/
+[Component controller]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/concepts/component_controller/
+[Components configuration language]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/config-language/components/
+[env]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/reference/stdlib/env/
+
+**Recommended reading**
+
+- [Components][]
+- [Components configuration language][]
+- [Component controller][]
+
+Components are the building blocks of a {{< param "PRODUCT_NAME" >}} configuration. They are configured and linked to create pipelines that collect, process, and output your telemetry data. Components are configured with `Arguments` and have `Exports` that may be referenced by other components.
+
+Let's look at a simple example pipeline:
+
+```river
+local.file "example" {
+    path = env("HOME") + "file.txt"
+}
+
+prometheus.remote_write "local_prom" {
+    endpoint {
+        url = "http://localhost:9090/api/v1/write"
+
+        basic_auth {
+            username = "admin"
+            password = local.file.example.content
+        }
+    }
+}
+```
+
+{{% admonition type="note" %}}
+[Component reference]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/reference/components/
+
+A list of all available components can be found in the [Component reference][]. Each component has a link to its documentation, which contains a description of what the component does, its arguments, its exports, and Example(s).
+{{% /admonition %}}
+
+This pipeline has two components: `local.file` and `prometheus.remote_write`. The `local.file` component is configured with a single argument, `path`, which is set by calling the [env][] standard library function to retrieve the value of the `HOME` environment variable and concatenating it with the string `"file.txt"`. The `local.file` component has a single export, `content`, which contains the contents of the file.
+
+The `prometheus.remote_write` component is configured with an `endpoint` block, containing the `url` attribute and a `basic_auth` block. The `url` attribute is set to the URL of the Prometheus remote write endpoint. The `basic_auth` block contains the `username` and `password` attributes, which are set to the string `"admin"` and the `content` export of the `local.file` component, respectively. The `content` export is referenced by using the syntax `local.file.example.content`, where `local.file.example` is the fully qualified name of the component (the component's type + its label) and `content` is the name of the export.
+
+<p align="center">
+<img src="/media/docs/agent/diagram-flow-by-example-basic-0.svg" alt="Flow of example pipeline with local.file and prometheus.remote_write components" width="200" />
+</p>
+
+{{% admonition type="note" %}}
+The `local.file` component's label is set to `"example"`, so the fully qualified name of the component is `local.file.example`. The `prometheus.remote_write` component's label is set to `"local_prom"`, so the fully qualified name of the component is `prometheus.remote_write.local_prom`.
+{{% /admonition %}}
+
+This example pipeline still doesn't do anything, so let's add some more components to it.
+
+## Shipping our first metrics
+
+[prometheus.exporter.unix]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/reference/components/prometheus.exporter.unix/
+[prometheus.scrape]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/reference/components/prometheus.scrape/
+[prometheus.remote_write]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/reference/components/prometheus.remote_write/
+
+**Recommended reading**
+
+- Optional: [prometheus.exporter.unix][]
+- Optional: [prometheus.scrape][]
+- Optional: [prometheus.remote_write][]
+
+Let's make a simple pipeline with a `prometheus.exporter.unix` component, a `prometheus.scrape` component to scrape it, and a `prometheus.remote_write` component to send the scraped metrics to Prometheus.
+
+```river
+prometheus.exporter.unix "localhost" { 
+    // This component exposes a lot of metrics by default, so we will keep all of the default arguments.
+}
+
+prometheus.scrape "default" {
+    // Setting the scrape interval lower to make it faster to be able to see the metrics
+    scrape_interval = "10s"
+
+    targets    = prometheus.exporter.unix.localhost.targets
+    forward_to = [
+        prometheus.remote_write.local_prom.receiver,
+    ]
+}
+
+prometheus.remote_write "local_prom" {
+    endpoint {
+        url = "http://localhost:9090/api/v1/write"
+    }
+}
+```
+
+Run the agent with:
+
+```bash
+/path/to/agent run config.river
+```
+
+Navigate to [http://localhost:3000/explore](http://localhost:3000/explore) in your browser. After ~15-20 seconds, you should be able to see the metrics from the `prometheus.exporter.unix` component! Try querying for `node_memory_Active_bytes` to see the active memory of your host.
+
+<p align="center">
+<img src="/media/docs/agent/screenshot-flow-by-example-memory-usage.png" alt="Screenshot of node_memory_Active_bytes query in Grafana" />
+</p>
+
+## Visualizing the relationship between components
+
+Let's look at an example pipeline:
+
+<p align="center">
+<img src="/media/docs/agent/diagram-flow-by-example-full-0.svg" alt="Flow of example pipeline with a prometheus.scrape, prometheus.exporter.unix, and prometheus.remote_write components" width="400" />
+</p>
+
+The above configuration defines three components:
+
+- `prometheus.scrape` - A component that scrapes metrics from components that export targets.
+- `prometheus.exporter.unix` - A component that exports metrics from the host, built around [node_exporter](https://github.com/prometheus/node_exporter).
+- `prometheus.remote_write` - A component that sends metrics to a Prometheus remote-write compatible endpoint.
+
+The `prometheus.scrape` component references the `prometheus.exporter.unix` component's targets export, which is a list of scrape targets. The `prometheus.scrape` component then forwards the scraped metrics to the `prometheus.remote_write` component.
+
+One rule is that components cannot form a cycle. This means that a component cannot reference itself directly or indirectly. This is to prevent infinite loops from forming in the pipeline.
+
+## Exercise for the reader
+
+[prometheus.exporter.redis]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/reference/components/prometheus.exporter.redis/
+
+**Recommended Reading**
+
+- Optional: [prometheus.exporter.redis][]
+
+Let's start a container running Redis and configure the agent to scrape metrics from it.
+
+```bash
+docker container run -d --name flow-redis -p 6379:6379 --rm redis
+```
+
+Try modifying the above pipeline to scrape metrics from the Redis exporter. You can refer to the [prometheus.exporter.redis][] component documentation for more information on how to configure it.
+
+To give a visual hint, you want to create a pipeline that looks like this:
+
+<p align="center">
+<img src="/media/docs/agent/diagram-flow-by-example-exercise-0.svg" alt="Flow of exercise pipeline, with a scrape, unix_exporter, redis_exporter, and remote_write component" width="600" />
+</p>
+
+{{% admonition type="note" %}}
+[concat]: https://grafana.com/docs/agent/<AGENT_VERSION>/flow/reference/stdlib/concat/
+
+You may find the [concat][] standard library function useful.
+{{% /admonition %}}
+
+You can run the agent with the new config file by running:
+
+```bash
+/path/to/agent run config.river
+```
+
+Navigate to [http://localhost:3000/explore](http://localhost:3000/explore) in your browser. After the first scrape, you should be able to query for `redis` metrics as well as `node` metrics!
+
+To shut down the Redis container, run:
+
+```bash
+docker container stop flow-redis
+```
+
+If you get stuck, you can always view a solution here:
+{{< collapse title="Solution" >}}
+
+```river
+// Configure your first components, learn about the standard library, and learn how to run the Agent!
+
+// prometheus.exporter.redis collects information about Redis and exposes
+// targets for other components to use
+prometheus.exporter.redis "local_redis" {
+    redis_addr = "localhost:6379"
+}
+
+prometheus.exporter.unix "localhost" { }
+
+// prometheus.scrape scrapes the targets that it is configured with and forwards
+// the metrics to other components (typically prometheus.relabel or prometheus.remote_write)
+prometheus.scrape "default" {
+    // This is scraping too often for typical use-cases, but is easier for testing and demo-ing!
+    scrape_interval = "10s"
+
+    // Here, prometheus.exporter.redis.local_redis.targets refers to the 'targets' export
+    // of the prometheus.exporter.redis component with the label "local_redis".
+    //
+    // If you have more than one set of targets that you would like to scrape, you can use
+    // the 'concat' function from the standard library to combine them.
+    targets    = concat(prometheus.exporter.redis.local_redis.targets, prometheus.exporter.unix.localhost.targets)
+    forward_to = [prometheus.remote_write.local_prom.receiver]
+}
+
+// prometheus.remote_write exports a 'receiver', which other components can forward
+// metrics to and it will remote_write them to the configured endpoint(s)
+prometheus.remote_write "local_prom" {
+    endpoint {
+        url = "http://localhost:9090/api/v1/write"
+    }
+}
+
+```
+
+{{< /collapse >}}
+
+## Finishing up and next steps
+
+You might have noticed that running the agent with the above configurations created a directory called `data-agent` in the directory you ran the agent from. This directory is where components can store data, such as the `prometheus.exporter.unix` component storing its WAL (Write Ahead Log). If you look in the directory, do you notice anything interesting? The directory for each component is the fully-qualified name!
+
+If you'd like to store the data elsewhere, you can specify a different directory by supplying the `--storage.path` flag to the agent's run command, for example, `/path/to/agent run config.river --storage.path /etc/grafana-agent`. Generally, you will want to use a persistent directory for this, as some components may use the data stored in this directory to perform their function.
+
+In the next tutorial, we will look at how to configure the agent to collect logs from a file and send them to Loki. We will also look at using different components to process metrics and logs before sending them.
diff --git a/docs/sources/flow/tutorials/flow-by-example/get-started.md b/docs/sources/flow/tutorials/flow-by-example/get-started.md
@@ -0,0 +1,89 @@
+---
+aliases:
+- /docs/grafana-cloud/agent/flow/tutorials/flow-by-example/faq/
+- /docs/grafana-cloud/monitor-infrastructure/agent/flow/tutorials/flow-by-example/faq/
+- /docs/grafana-cloud/monitor-infrastructure/integrations/agent/flow/tutorials/flow-by-example/faq/
+- /docs/grafana-cloud/send-data/agent/flow/tutorials/flow-by-example/faq/
+canonical: https://grafana.com/docs/agent/latest/flow/tutorials/flow-by-example/faq/
+description: Getting started with Flow-by-Example Tutorials
+title: Get started
+weight: 10
+---
+
+## Who is this for?
+
+This set of tutorials contains a collection of examples that build on each other to demonstrate how to configure and use [{{< param "PRODUCT_NAME" >}}][flow]. It assumes you have a basic understanding of what {{< param "PRODUCT_ROOT_NAME" >}} is and telemetry collection in general. It also assumes a base level of familiarity with Prometheus and PromQL, Loki and LogQL, and basic Grafana navigation. It assumes no knowledge of {{< param "PRODUCT_NAME" >}} or River concepts.
+
+[flow]: https://grafana.com/docs/agent/latest/flow
+
+## What is Flow?
+
+Flow is a new way to configure {{< param "PRODUCT_NAME" >}}. It is a declarative configuration language that allows you to define a pipeline of telemetry collection, processing, and output. It is built on top of the [River](https://github.com/grafana/river) configuration language, which is designed to be fast, simple, and debuggable.
+
+## What do I need to get started?
+
+You will need a Linux or Unix environment with Docker installed. The examples are designed to be run on a single host so that you can run them on your laptop or in a VM. You are encouraged to follow along with the examples using a `config.river` file and experiment with the examples yourself.
+
+To run the examples, you should have a Grafana Agent binary available. You can follow the instructions on how to [Install Grafana Agent as a Standalone Binary](https://grafana.com/docs/agent/latest/flow/setup/install/binary/#install-grafana-agent-in-flow-mode-as-a-standalone-binary) to get a binary.
+
+## How should I follow along?
+
+You can use this docker-compose file to set up a local Grafana instance alongside Loki and Prometheus pre-configured as datasources. The examples are designed to be run locally, so you can follow along and experiment with them yourself.
+
+```yaml
+version: '3'
+services:
+  loki:
+    image: grafana/loki:2.9.0
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/local-config.yaml
+  prometheus:
+    image: prom/prometheus:v2.47.0
+    command:
+      - --web.enable-remote-write-receiver
+      - --config.file=/etc/prometheus/prometheus.yml
+    ports:
+      - "9090:9090"
+  grafana:
+    environment:
+      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+    entrypoint:
+      - sh
+      - -euc
+      - |
+        mkdir -p /etc/grafana/provisioning/datasources
+        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
+        apiVersion: 1
+        datasources:
+        - name: Loki
+          type: loki
+          access: proxy
+          orgId: 1
+          url: http://loki:3100
+          basicAuth: false
+          isDefault: false
+          version: 1
+          editable: false
+        - name: Prometheus
+          type: prometheus
+          orgId: 1
+          url: http://prometheus:9090
+          basicAuth: false
+          isDefault: true
+          version: 1
+          editable: false
+        EOF
+        /run.sh
+    image: grafana/grafana:latest
+    ports:
+      - "3000:3000"
+```
+
+After running `docker-compose up`, open [http://localhost:3000](http://localhost:3000) in your browser to view the Grafana UI.
+
+The tutorials are designed to be followed in order and generally build on each other. Each example explains what it does and how it works. They are designed to be run locally, so you can follow along and experiment with them yourself.
+
+The Recommended Reading sections in each tutorial provide a list of documentation topics. To help you understand the concepts used in the example, read the recommended topics in the order given.