Skip to content

Commit

Permalink
fix: improve monitoring guide
Browse files Browse the repository at this point in the history
  • Loading branch information
wrussell1999 committed Nov 1, 2024
1 parent 90db7f8 commit 4bb3cf2
Show file tree
Hide file tree
Showing 2 changed files with 36 additions and 9 deletions.
45 changes: 36 additions & 9 deletions content/docs/15.how-to-guides/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,16 @@ scrape_configs:
- targets: ["<kestra-host-ip-address>:8081"]
```
Be sure to put the appropriate `<kestra-host-ip-address>` in the last line. Now, we will start the Prometheus server using the following docker command:
Be sure to put the appropriate `<kestra-host-ip-address>` in the last line, e.g. `localhost:8081` or `host.docker.internal:8081`.

::alert{type="info"}
If you're running everything in Docker on the same machine, you will need to change your host address to `host.docker.internal` rather than localhost.
::

We can start the Prometheus server using the following docker command:

```sh
docker run -d -p 9090:9090 -v ~/Documents/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
docker run -d -p 9090:9090 -v ./prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
```

You can now go to `http://localhost:9090/graph` and try out visualizing some metrics using the PromQL. Here is one of the graphs for `kestra_executor_execution_started_count_total` metric:
Expand All @@ -61,20 +67,41 @@ Let us now move on to setting up Grafana. You start by installing Grafana using
docker run -d -p 3000:3000 --name=grafana grafana/grafana-enterprise
```

You can open the Grafana server at `http://localhost:3000`. The default credentials are `admin` as both username and passowrd. Once logged into Grafana, click on the hamburger menu on the top left and go to Connections -> Data Sources.
You can open the Grafana server at `http://localhost:3000`. The default credentials are `admin` as both username and password. Once logged into Grafana, click on the hamburger menu on the top left and go to **Connections -> Data Sources**.

![grafana_data_sources](/docs/how-to-guides/setup-monitoring/grafana_data_sources.png)

Click on `Add new Data Source` button present on the top right, and select `Prometheus` from the time series databases list. In the `Prometheus server URL` text box, put in the following URL: `http://<host-ip-address>:9090`. All the other confoguration can be left as default. You can click on `Save and Test` button at the bottom, and confirm that the connection to Prometheus database is successful.
### Add Data Source

Click on **Add new Data Source** button present on the top right, and select **Prometheus** from the time series databases list. In the **Prometheus server URL** text box, put in the following URL: `http://<host-ip-address>:9090`. All the other configuration can be left as default. You can click on **Save and Test** button at the bottom, and confirm that the connection to Prometheus database is successful.

## Add Dashboard

We are now all set to create the Grafana dashboard. For this, click on the **+** button on the top of the page to add a **New Dashboard** to Grafana. Save the dashboard with an appropriate name. Then, click on **Add visualization**, and select **prometheus** as the data source.

We will create a Gauge that shows the number of tasks that are presently running. For this, select **Gauge** as the Visualization in the top right corner. In the PromQL metrics explorer text box, you can write `sum(kestra_worker_running_count)`. Click on **Run queries** button to ensure the Gauge shows up the number.

Head back to Kestra and create a number of tasks that will execute for a long time. The example below will sleep for 60 seconds:

```yaml
id: sleep
namespace: company.team
tasks:
- id: sleep_task
type: io.kestra.plugin.scripts.shell.Commands
commands:
- sleep 60
```

We are now all set to create the Grafana dashboard. For this, click on the `+` button on the top of the page to add a `New Dashboard` to Grafana. Save the dashboard with an appropriate name. Then, click on `Add visualization`, and select `prometheus` as the data source.
Now we have some long-running tasks in progress, we can check that the Gauge correctly reflects the count. You can now put an appropriate title in the Panel options that says **Tasks running**.

We will create a Gauge that shows the number of tasks that are presently running. For this, select `Guage` as the Visualization in the top right corner. In the PromQL metrics explorer text box, you can write `sum(kestra_worker_running_count)`. Click on `Run queries` button to ensure the Guage shows up the number. You can now run some long-running task, and check that the Guage correctly reflects the count. You can now put an appropriate title in the Panel options, say `Tasks running`. This is how your Grafana should look like:
This is how your Grafana should look like:

![grafana_tasks_running_guage](/docs/how-to-guides/setup-monitoring/grafana_tasks_running_guage.png)
![grafana_tasks_running_gauge](/docs/how-to-guides/setup-monitoring/grafana_tasks_running_gauge.png)

Click on `Save` and `Apply` to add this guage to the dashboard.
Click on **Save** and **Apply** to add this gauge to the dashboard.

Similarly, you can now keep on adding more graphs to your dashboard. Here is one of the sample dashboards for Kestra metrics.
Similarly, you can now keep on adding more graphs to your dashboard. Here is one of the example dashboards for Kestra metrics.

![kestra_metrics_dashboard](/docs/how-to-guides/setup-monitoring/kestra_metrics_dashboard.png)

0 comments on commit 4bb3cf2

Please sign in to comment.