OPENTELEMETRY TRACES AND METRICS

This repository gives you simple example on how you can move from Datadog traces and metrics (using statsd) with fully OpenTelemetry SDK. The OpenTelemetry collector agent should be installed as the replacement of Datadog Agent.

Run Only Application

First, prepare the DATADOG_AGENT_HOST and OTEL_EXPORTER_OTLP_ENDPOINT in the environment variable.

Then, run the application.

DATADOG_AGENT_HOST=xxx DATADOG_AGENT_HOST=yyy docker-compose -f docker-compose-app-only.yaml up --build --force-recreate

Run All Docker Containers

MY_POD_IP=$(ipconfig getifaddr en0) \
  NODE_NAME=my-local-k8s-node \
  DD_API_KEY=xxx \
  DOCKER_USER="$(id -u):$(id -g)" \
  PROMETHEUS_WRITE_ENDPOINT="http://grafana-mimir.example.com/api/v1/push" docker-compose up --build --force-recreate

What is the environment used for?

• MY_POD_IP is the IP of the pod where the application is running.
• NODE_NAME is the name of the node where the application is running. This is used to identify the node in the traces.
• DD_API_KEY is the Datadog API key. This is used to send traces and metrics to Datadog.
• DOCKER_USER is the user and group that should be used to run the application. This is used to avoid permission issues when writing to the log file.
• PROMETHEUS_WRITE_ENDPOINT is the Prometheus Write endpoint (or Pushgateway) where the metrics should be sent. This is used to send metrics to Prometheus. cfs-staging-alicloud-beta.sh All of these environment is not needed by the application, but only for Datadog Agent and OpenTelemetry Colletor Agent.

What Metrics Emitted By The Application

dd-sdk

The application emits the following metrics:

• poc_dd_sdk_statsd.login.success: The number of successful login requests.
• poc_dd_sdk_statsd.login.failure: The number of failed login requests. With the tags reason.

otel-sdk

Note, Grafana Mimir uses _ and not ., so poc_otel_sdk.login.success will become poc_otel_sdk_login_success as metric name in the Grafana Mimir.

The application emits the following metrics:

• poc_otel_sdk.login.success: The number of successful login requests.
• poc_otel_sdk.login.failure: The number of failed login requests. With the tags failure_reason.
• poc_otel_sdk.http_server_requests_total: The number of HTTP requests. With the tags http.method, http.route, and http.status.
• poc_otel_sdk.http_server_request_duration_ms: The duration of the HTTP request. With the tags http.method, http.route, and http.status.

But, the OpenTelemetry Library also emits the following metrics:

https://github.com/open-telemetry/opentelemetry-go-contrib/blob/v1.32.0/instrumentation/net/http/otelhttp/internal/semconv/v1.20.0.go#L81-L85

• http.server.request.size
• http.server.response.size
• "http.server.duration

Demo

Supposed you already have installed Datadog Agent and OpenTelemetry Collector Agent in the same cluster, and:

DATADOG_AGENT_HOST=datadog-apm.datadog.svc
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector.datadog.svc:4317

Then, run the application

kubectl create namespace otel-demo
kubectl apply -n otel-demo -f kubernetes-pod.yaml

Then port-forward the application to your local machine

kubectl port-forward -n otel-demo pod/demo-otel-collector-dd-sdk 8081:8081
kubectl port-forward -n otel-demo pod/demo-otel-collector-otel-sdk 8082:8082

If you want to port forward to your local IP:

kubectl port-forward -n otel-demo pod/demo-otel-collector-dd-sdk 8081:8081 --address 192.168.1.34

kubectl port-forward -n otel-demo pod/demo-otel-collector-otel-sdk 8082:8082 --address 192.168.1.34

Then, open the browser and access the application:

# dd-sdk
curl http://localhost:8081

# otel-sdk
curl http://localhost:8082

Other endpoint is login:

# dd-sdk
curl -X POST http://localhost:8081/login -d '{"username": "user1", "password": "password1"}'

# otel-sdk
curl -X POST http://localhost:8082/login -d '{"username": "user1", "password": "password1"}'

Try using k6:

docker run --platform linux/amd64 --network host --rm -i -v $(pwd)/k6:/k6 docker.io/grafana/k6:0.55.0 run /k6/login.js

Prometheus

If you have Prometheus installed, you can add /metrics endpoint on the Prometheus. Set OTLP_METRIC_HTTP_ENABLED=false to ensure you only emit metric via the Prometheus only. But, this is not recommended, because you will lose the ability to send the metrics to the OpenTelemetry Collector Agent (the Datadog metric won't be sent).

demo-otel-collector-otel-sdk.<namespace>.svc/metrics