[exporter/prometheus] Shutdown method does not shutdown server #35464
Labels
Comments
Argannor
added
bug
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Good find. I'll follow-up on the PR |
andrzej-stencel
pushed a commit
that referenced
this issue
Oct 30, 2024
**Description:** Shutdown the http.Server instance on exporter shutdown **Link to tracking Issue:** #35464 **Testing:** Manual testing. I included this version of the exporter in our internal distribution and deployed it to verify the metrics no longer go stale after receiving a `NOHUP` signal. **Documentation:** n/a --------- Co-authored-by: David Ashpole <[email protected]>
zzhlogin
pushed a commit
to zzhlogin/opentelemetry-collector-contrib-aws
that referenced
this issue
Nov 12, 2024
**Description:** Shutdown the http.Server instance on exporter shutdown **Link to tracking Issue:** open-telemetry#35464 **Testing:** Manual testing. I included this version of the exporter in our internal distribution and deployed it to verify the metrics no longer go stale after receiving a `NOHUP` signal. **Documentation:** n/a --------- Co-authored-by: David Ashpole <[email protected]>
|
Fixed by #35465 |
sbylica-splunk
pushed a commit
to sbylica-splunk/opentelemetry-collector-contrib
that referenced
this issue
Dec 17, 2024
**Description:** Shutdown the http.Server instance on exporter shutdown **Link to tracking Issue:** open-telemetry#35464 **Testing:** Manual testing. I included this version of the exporter in our internal distribution and deployed it to verify the metrics no longer go stale after receiving a `NOHUP` signal. **Documentation:** n/a --------- Co-authored-by: David Ashpole <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
exporter/prometheus
What happened?
Description
When the collector receives a
NOHUPsignal, it reloads the current configuration and calls theShutdownmethod. The Shutdown method of the prometheusexporter however does only close the listener, not thehttp.Server. This means that the old server does not accept any new connections, but all existing connections may continue to live forever. If a client (like Prometheus) keeps the connection between scrapes alive, the requests will be served by the old instance, meaning the metrics will no longer get any updates until they disappear. This is especially difficult to debug, since requests usingcurlorwgetwill be served by the new instance, seeing completely different metrics.Steps to Reproduce
Expected Result
Actual Result
I will provide a Pull Request for this issue
Collector version
v0.109.0/v0.110.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
No response
Log output
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: