monitoring: tweak search request duration metric to record success / failure

[ggilmore]

This tweaks the search request duration metric to record whether or not the search itself was successful. This follows the advice from Google SRE's Monitoring Distributed Systems:

Latency

... It’s important to distinguish between the latency of successful requests and the latency of failed requests.

For example, an HTTP 500 error triggered due to loss of connection to a database or other critical backend might be served very quickly; however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors.

[jtibshirani]

Took a quick look as I'm trying to learn more about our observability features.

[jtibshirani]

I'm curious how prometheus handles changes to metric types. Maybe because they're both histograms, it's not really a change and now there will just be labels?

[ggilmore]

@jtibshirani

In Prometheus' data model, time series are uniquely identified by their name (zoekt_search_duration_seconds) and any attached labels (success). In my interpretation, this means that adding a new label creates a new time series. When we write Grafana dasbhoards that query on zoekt_search_duration_seconds{success=true}, the old data points (from zoekt_search_duration_seconds observations that occurred before we added the success label ) won't be returned from the Prometheus server.

Does this answer your question?

[jtibshirani]

Yep, thanks for the reference.