This repository contains information on how to start with IoT using Arduino, Grafana, Prometheus and Grafana Loki. In this repository, we will use Grafana Cloud that includes hosted Grafana, Prometheus and Loki to remove the overhead of installing and maintaining these systems. All of these projects are open source and therefore if you want, you can run them by yourself.
- Setting up and using Arduino IDE
- Setting up and using Grafana Cloud
- Sending Metrics
- Sending Logs
- Using Loki and Prometheus libraries
- Examples of projects built with Arduino and Grafana Cloud
The open source Arduino IDE makes it easy to write code and upload it to the board. The Arduino software is easy to use for beginners, yet flexible enough for advanced users.
To start, we are going to install Arduino IDE, a platform that’s used to write and upload programs to Arduino compatible boards. Follow the installing instruction for your OS.
This step is not necessary if Arduino hardware is used. If you decide to use ESP32 development boards (we are using ESP32 boards in most of the projects listed below), we will need to do couple additional steps:
-
If your OS won’t recognize the USB serial automatically, you’ll probably need to install CP210x USB to UART Bridge VCP Driver that lets your computer communicate with your development board.
-
Add ESP32 boards definition that adds support for the EP32 board. Go to Arduino > Preferences and add url https://dl.espressif.com/dl/package_esp32_index.json to the Additional Boards Manager URLs input. This open-source board definition adds support for programming ESP32 boards.
Then go to Tools > Boards > Boards Manager and add ESP32 board manager.
Now, in Tools > Boards, you are able to select your ESP32 board. This tells the Arduino IDE which profile and base libraries to use when compiling the firmware image, and how to flash it to the board.
In the Arduino environment, the programs that we are writing are called sketches. Before uploading the sketches to your development board, make sure that:
- Sketch can be compiled. Click on the Verify button on top of IDE. Verifying goes through your sketch, checks for errors and compiles it.
- In the Tools > Board submenu, you’ve selected the correct type of development port
- In the Tools > Port submenu, you’ve selected the new correct COM port
- In the Tools > Board submenu, you’ve selected the correct type of development port
Sketches are then uploaded to development board using Upload button on top of IDE.
As mentioned, we are going to be using Grafana Cloud — which comes with hosted Grafana, Grafana Loki, and Graphite — for our data storage and data visualization. The free tier comes with 10,000 series for Prometheus metrics and 50GB for logs in Loki, which is definitely more than enough for simple monitoring solutions.
To start, we'll visit Grafana Cloud signup and create a new account. As soon as the account is all set up, we can see the portal with hosted Grafana, Loki, and Prometheus instances.
To setup your project for sennding metrics, click on the Send Metrics button
From the next page, first you will want to copy the Remote Write Endpoint address
And put the value in your config.h
#define GC_URL "prometheus-blocks-prod-us-central1.grafana.net"
Next copy the Username / Instance ID
#define GC_USER "137822"
Then create an API key by clicking Generate now
Give the key a meaingful name and choose MetricsPublisher
Then click Create API Key
Copy the value into the GC_PASS field
#define GC_PASS "eyJrIjoiMTkzNDFkMzM2YTNhZTRlNmE4ZDkyMjgzSTBhNGFiYTcwY2VjMzVjNiIsIm4iOiJlc3AzMi10ZXN0LTMiLCJpZCI6NDIwMDY1fQ=="
The final result should look something like:
#define WIFI_SSID "your_ssid"
#define WIFI_PASSWORD "your_wifi_password"
#define GC_URL "prometheus-blocks-prod-us-central1.grafana.net"
#define GC_PATH "/api/prom/push"
#define GC_PORT 443
#define GC_USER "137822"
#define GC_PASS "eyJrIjoiMTkzNDFkMzM2YTNhZTRlNmE4ZDkyMjgzSTBhNGFiYTcwY2VjMzVjNiIsIm4iOiJlc3AzMi10ZXN0LTMiLCJpZCI6NDIwMDY1fQ=="
Similar to metrics, click the Sending Logs button
We will take some values from the Data Source Settings section starting with the URL
#define GC_URL "logs-prod-us-central1.grafana.net"
Next grab the User
#define GC_USER "8435"
And now generate an API key by clicking Generate now
Create a meaninful name and choose MetricsPublisher
The final config.h should look similar to this
#define WIFI_SSID "your_ssid"
#define WIFI_PASSWORD "your_wifi_pass"
#define GC_URL "logs-prod-us-central1.grafana.net"
#define GC_PORT 443
#define GC_PATH "/loki/api/v1/push"
#define GC_USER "8435"
#define GC_PASS "eyJrIjoiMTkzNDFkMzM2YTNhZTRlNmE4ZDkyMjgzSTBhNGFiYTcwY2VjMzVjNiIsIm4iOiJlc3AzMi10ZXN0LTMiLCJpZCI6NDIwMDY1fQ=="
At this point, or anytime in the future, we can create the API keys for Loki and Prometheus, to publish metrics from the monitoring system to these databases. The API key can be created by clicking on API Keys in the navigation on the left side. Then we click on + Add API Key and create API keys.
Hosted Loki & Prometheus instances are automatically added as data sources in your hosted Grafana. You can find them under the name grafanacloud-NAME-logs and grafanacloud-NAME-prom. You can use these data sources with data from monitoring solutions to create dahsboards or use them in Explore.
We have created following Arduino libraries to make it easier for you to send data to Prometheus and Loki:
- https://github.com/grafana/prometheus-arduino
- https://github.com/grafana/loki-arduino
- https://github.com/grafana/arduino-prom-loki-transport
- https://github.com/grafana/arduino-snappy-proto
All libraries can be download and used directly trough your Arduino IDE Library manager.
Start with opening Arduino IDE. Then go to Tools > Manage Libraries...
Search for "Prometheus" and install PrometheusArduino, PromLokiTransport, SnappyProto (You may be prompted to install additional libraries, saying yes will be the easiest path but you can also install the necessary libraries manually, see below)
Search for "Loki" and install GrafanaLoki
For list of additional libraries that you might need to install, refer to PrometheusArduino dependencies and GrafanaLoki dependencies
For how to use PrometheusArduino and GrafanaLoki, refer to their READMEs and examples folder
We have created couple of DIY IoT projects that you can easily create by yourself. These project have README that contains all information needed for you to re-create them.