Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

statsd receiver does not display well for statsd_type timing when observed as observer_type summary #29508

Closed
charz opened this issue Nov 27, 2023 · 7 comments
Closed

statsd receiver does not display well for statsd_type timing when observed as observer_type summary #29508

charz opened this issue Nov 27, 2023 · 7 comments
Labels
bug Something isn't working closed as inactive receiver/statsd statsd related issues Stale

Comments

@charz
Copy link

charz commented Nov 27, 2023

Component(s)

receiver/statsd

What happened?

Description

The statsd receiver does not display well for statsd_type timing when observed as observer_type summary. I observe that <name>_timing_count is consistently 1, and <name>_timing_sum is incorrect.

Steps to Reproduce

The configuration

receivers:
  statsd:
    endpoint: "0.0.0.0:8125" # default
    aggregation_interval: 10s  # default
    enable_metric_type: false  # default
    is_monotonic_counter: true # default
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "histogram"
        histogram:
          max_size: 50
      - statsd_type: "timing"
        observer_type: "summary"
      - statsd_type: "distribution"
        observer_type: "histogram"
        histogram:
          max_size: 50


exporters:
  prometheus:
    endpoint: "0.0.0.0:9105"
    send_timestamps: false
    metric_expiration: 120m
    enable_open_metrics: false
    add_metric_suffixes: false
    resource_to_telemetry_conversion:
      enabled: true

processors:
  batch:

service:
  telemetry:
    logs:
      level: debug
    metrics:
      address: ":8889"
  pipelines:
    metrics:
     receivers: [statsd]
     exporters: [prometheus]

The statsd metrics from Ncat

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; date
Mon Nov 27 06:46:47 UTC 2023
Mon Nov 27 06:46:47 UTC 2023

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; date
Mon Nov 27 06:47:09 UTC 2023
Mon Nov 27 06:47:09 UTC 2023

Expected Result

Here are the expected results from statsd_exporter. I'm unsure why test_metric_ts_count in the statsd receiver is consistently 1.

The same test agaist statsd_exporter

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; date
Mon Nov 27 06:54:58 UTC 2023
Mon Nov 27 06:54:58 UTC 2023

$ curl -sS 127.0.0.1:9102/metrics |grep test.metric.ts; date
# HELP test_metric_ts Metric autogenerated by statsd_exporter.
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0.5"} 0.012
test_metric_ts{myKey="myVal",quantile="0.9"} 0.042
test_metric_ts{myKey="myVal",quantile="0.99"} 0.042
test_metric_ts_sum{myKey="myVal"} 0.05500000000000001
test_metric_ts_count{myKey="myVal"} 3
Mon Nov 27 06:55:00 UTC 2023

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; date
Mon Nov 27 06:55:06 UTC 2023
Mon Nov 27 06:55:06 UTC 2023

$ curl -sS 127.0.0.1:9102/metrics |grep test.metric.ts; date
# HELP test_metric_ts Metric autogenerated by statsd_exporter.
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0.5"} 0.012
test_metric_ts{myKey="myVal",quantile="0.9"} 0.042
test_metric_ts{myKey="myVal",quantile="0.99"} 0.042
test_metric_ts_sum{myKey="myVal"} 0.11
test_metric_ts_count{myKey="myVal"} 6
Mon Nov 27 06:55:09 UTC 2023

Actual Result

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; date
Mon Nov 27 06:46:47 UTC 2023
Mon Nov 27 06:46:47 UTC 2023

$ curl -sS 127.0.0.1:9105/metrics; date
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 1
test_metric_ts{myKey="myVal",quantile="0.1"} 1
test_metric_ts{myKey="myVal",quantile="0.5"} 1
test_metric_ts{myKey="myVal",quantile="0.9"} 1
test_metric_ts{myKey="myVal",quantile="0.95"} 1
test_metric_ts{myKey="myVal",quantile="1"} 1
test_metric_ts_sum{myKey="myVal"} 1
test_metric_ts_count{myKey="myVal"} 1
Mon Nov 27 06:46:59 UTC 2023

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; date
Mon Nov 27 06:47:09 UTC 2023
Mon Nov 27 06:47:09 UTC 2023

$ curl -sS 127.0.0.1:9105/metrics; date
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 1
test_metric_ts{myKey="myVal",quantile="0.1"} 1
test_metric_ts{myKey="myVal",quantile="0.5"} 1
test_metric_ts{myKey="myVal",quantile="0.9"} 1
test_metric_ts{myKey="myVal",quantile="0.95"} 1
test_metric_ts{myKey="myVal",quantile="1"} 1
test_metric_ts_sum{myKey="myVal"} 1
test_metric_ts_count{myKey="myVal"} 1
Mon Nov 27 06:47:17 UTC 2023

$ curl -sS 127.0.0.1:9105/metrics; date
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 1
test_metric_ts{myKey="myVal",quantile="0.1"} 1
test_metric_ts{myKey="myVal",quantile="0.5"} 1
test_metric_ts{myKey="myVal",quantile="0.9"} 1
test_metric_ts{myKey="myVal",quantile="0.95"} 1
test_metric_ts{myKey="myVal",quantile="1"} 1
test_metric_ts_sum{myKey="myVal"} 1
test_metric_ts_count{myKey="myVal"} 1
Mon Nov 27 06:47:25 UTC 2023

$ curl -sS 127.0.0.1:9105/metrics; date
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 1
test_metric_ts{myKey="myVal",quantile="0.1"} 1
test_metric_ts{myKey="myVal",quantile="0.5"} 1
test_metric_ts{myKey="myVal",quantile="0.9"} 1
test_metric_ts{myKey="myVal",quantile="0.95"} 1
test_metric_ts{myKey="myVal",quantile="1"} 1
test_metric_ts_sum{myKey="myVal"} 1
test_metric_ts_count{myKey="myVal"} 1
Mon Nov 27 06:47:36 UTC 2023

Collector version

v0.88.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  statsd:
    endpoint: "0.0.0.0:8125" # default
    aggregation_interval: 10s  # default
    enable_metric_type: false  # default
    is_monotonic_counter: true # default
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "histogram"
        histogram:
          max_size: 50
      - statsd_type: "timing"
        observer_type: "summary"
      - statsd_type: "distribution"
        observer_type: "histogram"
        histogram:
          max_size: 50


exporters:
  prometheus:
    endpoint: "0.0.0.0:9105"
    send_timestamps: false
    metric_expiration: 120m
    enable_open_metrics: false
    add_metric_suffixes: false
    resource_to_telemetry_conversion:
      enabled: true

processors:
  batch:

service:
  telemetry:
    logs:
      level: debug
    metrics:
      address: ":8889"
  pipelines:
    metrics:
     receivers: [statsd]
     exporters: [prometheus]

Log output

$ ./otelcol-contrib --config otel.yaml; date
2023-11-27T06:46:44.441Z	info	[email protected]/telemetry.go:84	Setting up own telemetry...
2023-11-27T06:46:44.441Z	info	[email protected]/telemetry.go:201	Serving Prometheus metrics	{"address": ":8889", "level": "Basic"}
2023-11-27T06:46:44.441Z	debug	[email protected]/exporter.go:273	Beta component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:44.441Z	debug	[email protected]/receiver.go:294	Beta component. May change in the future.	{"kind": "receiver", "name": "statsd", "data_type": "metrics"}
2023-11-27T06:46:44.446Z	info	[email protected]/service.go:143	Starting otelcol-contrib...	{"Version": "0.88.0", "NumCPU": 48}
2023-11-27T06:46:44.446Z	info	extensions/extensions.go:33	Starting extensions...
2023-11-27T06:46:44.446Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "exporter", "data_type": "metrics", "name": "prometheus", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-11-27T06:46:44.447Z	info	[email protected]/service.go:169	Everything is ready. Begin running and processing data.
2023-11-27T06:46:50.172Z	debug	[email protected]/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:50.172Z	debug	[email protected]/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.041Z	debug	[email protected]/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.041Z	debug	[email protected]/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.448Z	debug	[email protected]/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.448Z	debug	[email protected]/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.448Z	debug	[email protected]/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:59.666Z	debug	[email protected]/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:59.666Z	debug	[email protected]/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:59.666Z	debug	[email protected]/collector.go:386	metric served: Desc{fqName: "test_metric_ts", help: "", constLabels: {}, variableLabels: {myKey}}	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:14.448Z	debug	[email protected]/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:14.448Z	debug	[email protected]/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:14.448Z	debug	[email protected]/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:17.351Z	debug	[email protected]/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:17.351Z	debug	[email protected]/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:17.351Z	debug	[email protected]/collector.go:386	metric served: Desc{fqName: "test_metric_ts", help: "", constLabels: {}, variableLabels: {myKey}}	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}

2023-11-27T06:47:25.566Z	debug	[email protected]/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:25.566Z	debug	[email protected]/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:25.566Z	debug	[email protected]/collector.go:386	metric served: Desc{fqName: "test_metric_ts", help: "", constLabels: {}, variableLabels: {myKey}}	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:36.208Z	debug	[email protected]/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:36.208Z	debug	[email protected]/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:36.208Z	debug	[email protected]/collector.go:386	metric served: Desc{fqName: "test_metric_ts", help: "", constLabels: {}, variableLabels: {myKey}}	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}

^C2023-11-27T06:47:41.510Z	info	[email protected]/collector.go:250	Received signal from OS	{"signal": "interrupt"}
2023-11-27T06:47:41.510Z	info	[email protected]/service.go:178	Starting shutdown...
2023-11-27T06:47:41.510Z	debug	[email protected]/reporter.go:97	UDP Transport ([::]:8125) - ReadFrom error: read udp [::]:8125: use of closed network connection	{"kind": "receiver", "name": "statsd", "data_type": "metrics"}
2023-11-27T06:47:41.510Z	info	extensions/extensions.go:50	Stopping extensions...
2023-11-27T06:47:41.510Z	info	[email protected]/service.go:192	Shutdown complete.
Mon Nov 27 06:47:41 UTC 2023

Additional context

No response
@charz charz added bug Something isn't working needs triage New item requiring triage labels Nov 27, 2023
@github-actions github-actions bot added the receiver/statsd statsd related issues label Nov 27, 2023
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot mentioned this issue Nov 28, 2023
Closed
98 tasks
@atoulme atoulme removed the needs triage New item requiring triage label Nov 29, 2023
@atoulme
Copy link
Contributor

atoulme commented Nov 29, 2023

@charz thank you for this report, would you like to try to fix this issue?

@charz
Copy link
Author

charz commented Dec 5, 2023

@atoulme Do you have any guidelines that I should follow?

@atoulme
Copy link
Contributor

atoulme commented Dec 5, 2023

Sure, I'd take your tests and curl requests and turn this into a test that fails. From there, I'd open a PR to show the failing test and ask for help to resolve, or try to implement a fix.

@charz
Copy link
Author

charz commented Dec 27, 2023

Just figured it out. Will send a PR later.

➜  ~ echo "test.metric.ts:22|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:22|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:22|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:22|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;

➜  ~ curl -sS 127.0.0.1:9105/metrics
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 2
test_metric_ts{myKey="myVal",quantile="0.1"} 2
test_metric_ts{myKey="myVal",quantile="0.5"} 22
test_metric_ts{myKey="myVal",quantile="0.9"} 22
test_metric_ts{myKey="myVal",quantile="0.95"} 22
test_metric_ts{myKey="myVal",quantile="1"} 22
test_metric_ts_sum{myKey="myVal"} 140
test_metric_ts_count{myKey="myVal"} 10

➜  ~ curl -sS 127.0.0.1:9105/metrics
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 2
test_metric_ts{myKey="myVal",quantile="0.1"} 2
test_metric_ts{myKey="myVal",quantile="0.5"} 2
test_metric_ts{myKey="myVal",quantile="0.9"} 2
test_metric_ts{myKey="myVal",quantile="0.95"} 2
test_metric_ts{myKey="myVal",quantile="1"} 2
test_metric_ts_sum{myKey="myVal"} 6
test_metric_ts_count{myKey="myVal"} 3

charz pushed a commit to charz/opentelemetry-collector-contrib that referenced this issue Dec 28, 2023
Fix statsd receiver aggregate memtrics in various instruments.
997af76 
The receiver uses each metric's address as the key to create instruments,
but the port in the address of each metric is dynamically assigned.
This results in the StatsD receiver creating numerous instruments and
being unable to aggregate metrics together.

Fixes open-telemetry#29508, fixex open-telemetry#23809
This was referenced Dec 28, 2023
Closed
Closed
@github-actions GitHub Actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Feb 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working closed as inactive receiver/statsd statsd related issues Stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix statsd receiver aggregate metrics in various instruments. charz/opentelemetry-collector-contrib
2 participants
@atoulme @charz