Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/awscontainerinsight] Filter terminated pods from node request metrics. #27299

Merged
merged 12 commits into from
Dec 8, 2023

Conversation

jefchien
Copy link
Contributor

@jefchien jefchien commented Oct 2, 2023

Description: The node_<cpu|memory>_request metrics and metrics derived from them (node_<cpu|memory>_reserved_capacity) differ from the output of kubectl describe node <node_name>. This is because kubectl filters out terminated pods. See linked issue for more details.

Adds a filter for terminated (succeeded/failed state) pods.

Link to tracking Issue: #27262

Testing: Added unit test to validate pod state filtering. Built and deployed changes to cluster. Deployed cpu-test pod.

image

The gap is when the change was deployed. The metric drops after the deployment due to the filter. The metric can be seen spiking up while the cpu-test pod is running (~19:15) and then returns to the previous request size after it has terminated.

Documentation: N/A

@jefchien jefchien requested a review from Aneurysm9 as a code owner October 2, 2023 18:03
@jefchien jefchien requested a review from a team October 2, 2023 18:03
@github-actions github-actions bot requested a review from pxaws October 2, 2023 18:03
@jefchien jefchien changed the title Filter terminated pods from node request metrics. [receiver/awscontainerinsight] Filter terminated pods from node request metrics. Oct 2, 2023
Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added the Stale label Nov 10, 2023
@github-actions github-actions bot removed the Stale label Nov 15, 2023
@Aneurysm9
Copy link
Member

awscontainerinsightreceiver/internal/stores/podstore_test.go:632: File is not gofmt-ed with -s -r 'interface{} -> any' (gofmt)

Lint failure needs to be fixed.

@jefchien
Copy link
Contributor Author

awscontainerinsightreceiver/internal/stores/podstore_test.go:632: File is not gofmt-ed with -s -r 'interface{} -> any' (gofmt)

Lint failure needs to be fixed.

Yep. The linting must've changed at some point. Will update it.

@jefchien
Copy link
Contributor Author

The lint failure has been fixed.

@bryan-aguilar bryan-aguilar added the ready to merge Code review completed; ready to merge by maintainers label Nov 29, 2023
@TylerHelmuth TylerHelmuth merged commit 5449419 into open-telemetry:main Dec 8, 2023
88 checks passed
@github-actions github-actions bot added this to the next release milestone Dec 8, 2023
@jefchien jefchien deleted the filter-terminated-pods branch December 8, 2023 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready to merge Code review completed; ready to merge by maintainers receiver/awscontainerinsight
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants