[Bug]: Prometheus rule KubeletTooManyPods incorrect statistics

[jeffaryhe]

What happened?

prometheus-operator/kube-prometheus#2558 pls look this

Please provide any helpful snippets.

No response

What parts of the codebase are affected?

Rules

I agree to the following terms:

• I agree to follow this project's Code of Conduct.
• I have filled out all the required information above to the best of my ability.
• I have searched the issues of this repository and believe that this is not a duplicate.
• I have confirmed this bug exists in the default branch of the repository, as of the latest commit at the time of submission.

[skl]

Hi @jeffaryhe, thanks for the report. I had a look at the issue you raised:

• Prometheus rule KubeletTooManyPods incorrect statistics prometheus-operator/kube-prometheus#2558

I also had a look at the KubeletTooManyPods alert rule:

kubernetes-mixin/alerts/kubelet.libsonnet

Lines 54 to 63 in bad0615

	expr: |||
	count by(%(clusterLabel)s, node) (
	(kube_pod_status_phase{%(kubeStateMetricsSelector)s,phase="Running"} == 1) * on(instance,pod,namespace,%(clusterLabel)s) group_left(node) topk by(instance,pod,namespace,%(clusterLabel)s) (1, kube_pod_info{%(kubeStateMetricsSelector)s})
	)
	/
	max by(%(clusterLabel)s, node) (
	kube_node_status_capacity{%(kubeStateMetricsSelector)s,resource="pods"} != 1
	) > 0.95
	||| % $._config,
	'for': '15m',

From what I can see, the alert fires if the count of running pods on a node is at >95% that of the pod limit for that node.

Can you help me understand which statistics you see as incorrect? For example, do you think part of the alert rule could be improved?