Exportable service metrics

[j08lue]

Scope out what it would take to gather service health / usage metrics and make them accessible to any harvester of such metrics, e.g. to feed into a Grafana dashboard or OpenSearch.

Similar to what veda-backend has been doing, e.g.

• Implement OpenTelemetry in STAC & Raster APIs NASA-IMPACT/veda-backend#288

Perhaps related to

• Investigate How to Create Custom Metrics for Request Load #41

Acceptance criteria

• Describe possible pathways to an interface that exposes service health and usage metrics / traces

[j08lue]

Why this is coming up now:

I learned that the Resource Health building block in EOEPCA will possibly in the future be able to receive OpenTelemetry traces from other building blocks, to ease debugging.

VEDA Backend has set up a centralized usage metrics overview (dashboard), which would also be highly relevant as a feature for eoapi-k8s.

• OpenTelemetry implementation is validated and configured in veda-backend  NASA-IMPACT/veda-backend#349
• Grafana dashboards setup for fine grained STAC catalog analytics NASA-IMPACT/veda-backend#350

[ranchodeluxe]

I think we have about 75% of this already done in main b/c our metrics are already exportable via Prometheus on an API endpoint (that's how we have custom metrics about each service to autoscale by) and they get fed to Grafana. If I understand correctly the only addition here would be either:

1. setting up an OTeL deployment that feeds more data into Prometheus
2. setting up an OTeL exporter deployment that pushes data to an OTeL receiver service that EOEPCA+ sets up

That said we use ETOL on the fire atlas stuff and without writing custom metrics I haven't seen a lot of good data coming out of it that we can't get from other metrics APIs already. So something to think about

[j08lue]

Great to hear the infrastructure is basically already there and we just need to add some custom metrics.

Let us see once our services are up, which metrics that could be.

What is ETOL? 🙏

[ranchodeluxe]

What is ETOL? 🙏

The Encyclopedia of Trotskyism Online 😉 I mean OTeL

[ranchodeluxe]

@j08lue: It also occurred to me this means we'd have to build our own custom runtimes to add OTeL stuff 😞 I really, truly feel like this is yet another example of something that should be plumbed through and turned on in all the upstream libraries 😬

[j08lue]

I think we have about 75% of this already done in main b/c our metrics are already exportable via Prometheus on an API endpoint

We are now planning on including OpenTelemetry in our EOEPCA runtimes and have them feed into Prometheus. From there they can get exposed via Grafana or picked up by external observability services.

We should document the approach EOEPCA takes for other users of eoapi-k8s to follow.

We will discuss in the context of EOEPCA, which additional metrics (besides liveliness / uptime) we would like to gather.

• Provide more metrics EOEPCA/data-access#114