akka http.server.connection.open metric in histogram format?

[kjorg50]

Hi Kamon team,

I'm looking to better understand the http.server.connection.open metrics a little bit better for my akka-http application. I see on the docs that this metric is a "range sampler" which represents the "number of open connections", and that part is straightforward. However, when I use a prometheus exporter and view this metric in prometheus format, I don't quite understand the output, for example:

# HELP http_server_connection_open Number of open connections
# TYPE http_server_connection_open histogram
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="0.5",component="akka.http.server"} 1815.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="1.0",component="akka.http.server"} 1815.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="5.0",component="akka.http.server"} 1818.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="10.0",component="akka.http.server"} 2022.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="30.0",component="akka.http.server"} 2022.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="100.0",component="akka.http.server"} 2022.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="300.0",component="akka.http.server"} 2022.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="1000.0",component="akka.http.server"} 2022.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="3000.0",component="akka.http.server"} 2022.0
http_server_connection_open_bucket{port="8443",interface="0.0.0.0",le="+Inf",component="akka.http.server"} 2022.0
http_server_connection_open_count{port="8443",interface="0.0.0.0",component="akka.http.server"} 2022.0
http_server_connection_open_sum{port="8443",interface="0.0.0.0",component="akka.http.server"} 2043.0

Is the _count the total number of connections that have been opened since the application started up? What are the buckets used for in this case? Does it even make sense to plot this metric in histogram format?

Any insight is appreciated, thanks!

[pnerg]

I'd love an answer to this question as I'm also struggling to bend my head around how the range sampler metric is translated into a histogram.
I get that under the hood the http.server.connection.open metric builds on a range sampler

The range sampler internally tracks three variables: the current value, the minimum and the maximum observed value. These three values are read and stored in a histogram every 200 milliseconds (by default) and produce a value distribution on each reporting interval./metric-instruments

I'm struggling to understand how these distribution values are represented in the histogram.

[pnerg]

ping @SimunKaracic @ivantopo hoping you experts can help out

[pnerg]

Looking into the code of the Prometheus exporter I realised one can decide if one wants the metric displayed as a summary or histogram.
So I applied the config

  kamon.prometheus.summaries.metrics = ["*"]

The result in this case is that all distribution type metrics are displayed as summaries and not histograms.
So for http server related metrics one would see something like

# HELP http_server_request_active Number of requests being processed simultaneously at any point in time
# TYPE http_server_request_active summary
http_server_request_active{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.5"} 0.0
http_server_request_active{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.75"} 1.0
http_server_request_active{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.95"} 0.0
http_server_request_active{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.99"} 0.0
http_server_request_active{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.999"} 0.0
http_server_request_active_count{port="9696",interface="0.0.0.0",component="akka.http.server"} 2055.0
http_server_request_active_sum{port="9696",interface="0.0.0.0",component="akka.http.server"} 3.0
# HELP http_server_connection_open Number of open connections
# TYPE http_server_connection_open summary
http_server_connection_open{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.5"} 0.0
http_server_connection_open{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.75"} 2.0
http_server_connection_open{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.95"} 0.0
http_server_connection_open{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.99"} 0.0
http_server_connection_open{port="9696",interface="0.0.0.0",component="akka.http.server",quantile="0.999"} 0.0
http_server_connection_open_count{port="9696",interface="0.0.0.0",component="akka.http.server"} 2055.0
http_server_connection_open_sum{port="9696",interface="0.0.0.0",component="akka.http.server"} 1114.0

Not that it made more sense to me.
E.g. I ran three requests and got

http_server_request_active_count{port="9696",interface="0.0.0.0",component="akka.http.server"} 2055.0
http_server_request_active_sum{port="9696",interface="0.0.0.0",component="akka.http.server"} 3.0

The active_count keeps increasing even if I don't run a single request.
Don't get the quantiles either, exactly what do they represent for open connections?

Active requests is more of a gauge than a distribution don't see how this really plays out.
E.g. JVM threads are measured as a gauge not sure what use this distribution brings here.
I hoped someone from the Kamon team could explain this.

I'm going to work around at my end and generate my own metrics as these distribution types bring little value for this type of measurement. I get one can use histograms to measure e.g. latency but not for open connections.

[pnerg]

Finally I got around and made a PR #1011 that addresses the topic of this discussion

[SimunKaracic]

Here, hopefully this will clarify any questions you have about Range Samplers!
https://kamon.io/blog/monitoring-queues-and-resources-with-kamon-range-samplers/?utm_source=githubdiscussion