This repository gives you simple example on how you can move from Datadog traces and metrics (using statsd) with fully OpenTelemetry SDK. The OpenTelemetry collector agent should be installed as the replacement of Datadog Agent.
First, prepare the DATADOG_AGENT_HOST
and OTEL_EXPORTER_OTLP_ENDPOINT
in the environment variable.
Then, run the application.
DATADOG_AGENT_HOST=xxx DATADOG_AGENT_HOST=yyy docker-compose -f docker-compose-app-only.yaml up --build --force-recreate
MY_POD_IP=$(ipconfig getifaddr en0) \
NODE_NAME=my-local-k8s-node \
DD_API_KEY=xxx \
DOCKER_USER="$(id -u):$(id -g)" \
PROMETHEUS_WRITE_ENDPOINT="http://grafana-mimir.example.com/api/v1/push" docker-compose up --build --force-recreate
What is the environment used for?
MY_POD_IP
is the IP of the pod where the application is running.NODE_NAME
is the name of the node where the application is running. This is used to identify the node in the traces.DD_API_KEY
is the Datadog API key. This is used to send traces and metrics to Datadog.DOCKER_USER
is the user and group that should be used to run the application. This is used to avoid permission issues when writing to the log file.PROMETHEUS_WRITE_ENDPOINT
is the Prometheus Write endpoint (or Pushgateway) where the metrics should be sent. This is used to send metrics to Prometheus. cfs-staging-alicloud-beta.sh All of these environment is not needed by the application, but only for Datadog Agent and OpenTelemetry Colletor Agent.
The application emits the following metrics:
poc_dd_sdk_statsd.login.success
: The number of successful login requests.poc_dd_sdk_statsd.login.failure
: The number of failed login requests. With the tagsreason
.
Note, Grafana Mimir uses
_
and not.
, sopoc_otel_sdk.login.success
will becomepoc_otel_sdk_login_success
as metric name in the Grafana Mimir.
The application emits the following metrics:
poc_otel_sdk.login.success
: The number of successful login requests.poc_otel_sdk.login.failure
: The number of failed login requests. With the tagsfailure_reason
.poc_otel_sdk.http_server_requests_total
: The number of HTTP requests. With the tagshttp.method
,http.route
, andhttp.status
.poc_otel_sdk.http_server_request_duration_ms
: The duration of the HTTP request. With the tagshttp.method
,http.route
, andhttp.status
.
But, the OpenTelemetry Library also emits the following metrics:
http.server.request.size
http.server.response.size
"http.server.duration
Supposed you already have installed Datadog Agent and OpenTelemetry Collector Agent in the same cluster, and:
DATADOG_AGENT_HOST=datadog-apm.datadog.svc
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector.datadog.svc:4317
Then, run the application
kubectl create namespace otel-demo
kubectl apply -n otel-demo -f kubernetes-pod.yaml
Then port-forward the application to your local machine
kubectl port-forward -n otel-demo pod/demo-otel-collector-dd-sdk 8081:8081
kubectl port-forward -n otel-demo pod/demo-otel-collector-otel-sdk 8082:8082
If you want to port forward to your local IP:
kubectl port-forward -n otel-demo pod/demo-otel-collector-dd-sdk 8081:8081 --address 192.168.1.34
kubectl port-forward -n otel-demo pod/demo-otel-collector-otel-sdk 8082:8082 --address 192.168.1.34
Then, open the browser and access the application:
# dd-sdk
curl http://localhost:8081
# otel-sdk
curl http://localhost:8082
Other endpoint is login:
# dd-sdk
curl -X POST http://localhost:8081/login -d '{"username": "user1", "password": "password1"}'
# otel-sdk
curl -X POST http://localhost:8082/login -d '{"username": "user1", "password": "password1"}'
Try using k6
:
docker run --platform linux/amd64 --network host --rm -i -v $(pwd)/k6:/k6 docker.io/grafana/k6:0.55.0 run /k6/login.js
If you have Prometheus installed, you can add /metrics endpoint on the Prometheus.
Set OTLP_METRIC_HTTP_ENABLED=false
to ensure you only emit metric via the Prometheus only.
But, this is not recommended, because you will lose the ability to send the metrics to the OpenTelemetry Collector Agent (the Datadog metric won't be sent).
demo-otel-collector-otel-sdk.<namespace>.svc/metrics