Provide operational metrics in OpenTelemetry Collector

[tomkerkhove]

Proposal

Provide operational metrics, that we have for Prometheus today, but push them to an OpenTelemetry Collector through OTEL.

Use-Case

Allow end-users to choose which metrics platform to use, instead of only supporting Prometheus.

Status

• Feature implementation
• Provide docs
• Integrate in Helm chart

Anything else?

A proposal is open on #1281 to bring scaler metrics to OpenTelemetry Collector as well.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[tomkerkhove]

@ilyalexin this is the issue that we discussed as a good starting point for OTEL Collector integration.

This should be supported in both Operator & Metric Server and have parity with https://keda.sh/docs/2.9/operate/prometheus/.

[tomkerkhove]

Related: #3695

[ilaleksin]

In order to resolve implement this feature one need to import opentelemetry sdk
At this moment keda has indirect dependency to opentelemetry v0.20.0 through k8s/api-server v0.25.4

In order to use opentelemtry v1.0+ we need to update api-server to v0.26.0

This work is already in progress under #4107

[tomkerkhove]

@kedacore/keda-maintainers Do we have an agreement that this is a nice addition?

[JorTurFer]

yeah, I think that generating OpenTelemetry data is worth. In our case for example, we have to deploy a Prometheus server to scrape metrics from other services (like KEDA, nginx, etc) and our collector scrapes that prometheus server to send the telemetry to the final backend (I know that collector can scrape the services directly, but it doesn't support Prometheus operator CRDs).
If KEDA (and other services) supports OpenTelemetry Protocol, we can use it to send the metrics directly to the collector and route them from there as we want

[tomkerkhove]

💯 - I'll get this on our roadmap to contribute from Microsoft side!

[SpiritZhou]

Hi @tomkerkhove @JorTurFer ,

Here is my basic design for implementing this feature:

1. Users can select their preferred metrics collection mode (Prometheus Only/OpenTelemetry Only/Prometheus and OpenTelemetry Both) using environment variables. The Keda operator will read these configuration settings (PROMETHEUS_ENABLE, OPENTELEMETRY_ENABLE, and OPENTELEMETRY_ENDPOINT).
2. Refactor the current Prometheus.go file by adding a middleware struct, such as MetricLogger, to handle initialization and metrics collection for both Prometheus and OpenTelemetry.

Do you have any suggestions regarding this design?

[JorTurFer]

I agree with this implementation. It's more or less what I thought about it indeed.
The only thing that I'd change is that instead of using new env vars for OTEL, I prefer to use the predefined variables when it's possible: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/

[JorTurFer]

I've found this exporter that we could use for exporting logs too: https://github.com/agoda-com/opentelemetry-go/tree/main/otelzap