- Caddy Server already installed and functional
- Server using Debian (it's up to you to adapt the tutorial according to your distribution)
In order to monitor the data on our Caddy server, we need to activate metrics and logs.
Activating the Caddy metrics (by using its API) will enable Prometheus to collect the data. To do this, you need to modify the "Caddyfile":
sudo nano /etc/caddy/CaddyfileIt is necessary to add "metrics" to the global options and that the admin endpoint is not set to off. If you have the "admin off" command, this will disable Caddy's REST API and make it impossible for metrics to work.
{
servers {
metrics
}
}Restart the Caddy service:
sudo systemctl restart caddyCheck that the service is active:
sudo systemctl status caddyTo be able to monitor Caddy logs via Loki, you need to configure them correctly. To do this, I'll take the example of my configuration.
I have 3 log files to monitor:
- "caddy_main.log": my global log file
- "domain.com.log": My log file for my main domain
- "grafana.domain.com.log": My log file for my sub-domain.
These files are located in "/var/log/caddy".
To do this, add these lines to the global options in "Caddyfile":
{
log {
output file /var/log/caddy/caddy_main.log {
roll_size 100MiB
roll_keep 5
roll_keep_for 100d
}
format json
level INFO
}
}And the same goes for your domains/subdomains:
domaine.com {
log {
output file /var/log/caddy/domain.com.log {
roll_size 100MiB
roll_keep 5
roll_keep_for 100d
}
format json
level INFO
}
}
grafana.domaine.com {
log {
output file /var/log/caddy/grafana.domain.com.log {
roll_size 100MiB
roll_keep 5
roll_keep_for 100d
}
format json
level INFO
}
}Restart the Caddy service:
sudo systemctl restart caddyCheck that the service is active:
sudo systemctl status caddyInstalling Prometheus:
sudo apt update
sudo apt install prometheus prometheus-node-exporterModifying the configuration file:
sudo nano /etc/prometheus/prometheus.ymlAdded lines below "scrape_configs":
- job_name: caddy
static_configs:
- targets: ['localhost:2019']You should have a configuration similar to this:
# Sample config for Prometheus.
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'example'
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
#rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
scrape_timeout: 5s
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: node
# If prometheus-node-exporter is installed, grab stats about the local
# machine by default.
static_configs:
- targets: ['localhost:9100']
- job_name: caddy
static_configs:
- targets: ['localhost:2019']
Restart the Prometheus service:
sudo systemctl restart prometheusCheck that the service is active:
sudo systemctl status prometheusAdd Grafana packages:
# mkdir -p /etc/apt/keyrings/
# wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/keyrings/grafana.gpg
# echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee /etc/apt/sources.list.d/grafana.listInstall Loki and Promtail:
apt update
apt install loki promtailEdit the Promtail configuration file:
sudo nano /etc/promtail/config.ymlAdd the following lines at the end of the file:
- job_name: caddy
static_configs:
- targets:
- localhost
labels:
job: caddy
__path__: /var/log/caddy/*log
agent: caddy-promtail
pipeline_stages:
- json:
expressions:
duration: duration
status: status
- labels:
duration:
status:You should have a configuration similar to this:
# This minimal config scrape only single log file.
# Primarily used in rpm/deb packaging where promtail service can be started during system init process.
# And too much scraping during init process can overload the complete system.
# https://github.com/grafana/loki/issues/11398
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
#NOTE: Need to be modified to scrape any additional logs of the system.
__path__: /var/log/messages
- job_name: caddy
static_configs:
- targets:
- localhost
labels:
job: caddy
__path__: /var/log/caddy/*log
agent: caddy-promtail
pipeline_stages:
- json:
expressions:
duration: duration
status: status
- labels:
duration:
status:
Restart Promtail and Loki services:
sudo systemctl restart promtail
sudo systemctl restart lokiCheck that the services is active:
sudo systemctl status promtail
sudo systemctl status lokisudo apt-get install -y apt-transport-https software-properties-common wgetsudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/nullecho "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.listsudo apt updatesudo apt install grafana-enterprise
This Dashboard integrates Prometheus + Loki.
This is a V1 which is not finished and not verified.
It can nevertheless serve as a basis for making your own Dashoard
Dashboard link : https://grafana.com/grafana/dashboards/20802