Not able to proceed consolidation on big cluster

[Hawksbk]

Hey, we are faced with the issue that Karpenter not feeling well with consolidation on the big cluster. (~700 nodes):

And such behavior was together with consolidation timeouts:

Based on where timeout comes from: https://github.com/kubernetes-sigs/karpenter/blob/8d819c1ddaea095d4fa9abb53b20b536c0c45802/pkg/controllers/disruption/singlenodeconsolidation.go#L53-L52

Looks like Karpenter not able to proceed fast enough in case of the big queue of candidates for disruption.

What is bothering me as well, is that Karpenter runs on separate nodes with enough computing (GOMAXPROCS tooks right 8 cores).

But doesn't seem that it's trying to parallel operation much due to the usage of 3 cores being quite stable and not going higher:

---

We temporarily mitigated it first by moving karpenter to nodes with more performance CPU (c7g -> c8g), and limited nodepools to use bigger hosts to decrease the overall number of hosts, but it's a kind of temporary solution.

---

So following questions:

• Is it possible to make efforts to improve the performance of karpenter in a matter of parallel operation or any other things that can be done in such a situation?
• Any recommendations to better understand the limit of how close we are to the red line?

[nantiferov]

I think if this proposal #1733 will be implemented, it might help with cases like this.

[Hawksbk]

Just in case:

EKS version:

• v1.31.4-eks-2d5f260
• public.ecr.aws/karpenter/controller:1.1.1@sha256:fe383abf1dbc79f164d1cbcfd8edaaf7ce97a43fbd6cb70176011ff99ce57523

[jonathan-innis]

The other thing here is just the improvement of scheduling in general to be parallelized a bit so that we can use more cores. I'd agree that the timeout is probably something that we should either consider parameterizing or just removing entirely

[jonathan-innis]

/triage accepted

[jonathan-innis]

cc: @rschalo

I know that he has some thoughts on how we might be able to make this better

[jonathan-innis]

/priority important-longterm