Keda in GKE Autopilot can't connect to Metric Server

[Positronico]

Hi!

I've installed Keda 2.10.2 via Helm in a GKE Autopilot 1.25.7 and v1beta1.external.metrics.k8s.io status is False (FailedDiscoveryCheck).

Health details shows:

FailedDiscoveryCheck: failing or missing response from https://10.21.130.23:6443/apis/external.metrics.k8s.io/v1beta1: Get "https://10.21.130.23:6443/apis/external.metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

First troubleshooting topic in the docs is exactly "Kubernetes Control plane is unable to communicate to Metric server?" and suggests a problem in clusters with CNI like Cilium, which is the case here.

But the proposed solution is to enable hostNetwork, which is not allowed in the autopilot due to hostPort restrictions.

Does anyone have a GKE autopilot running Keda?

[saez0pub]

I suspect a network firewall issue.
But, as autopilot nodes firewall rules are managed by google, I'm searching a way to allow it.

[saez0pub]

You have to allow the control plane IP network to access to node tag on port 6443

[Positronico]

We ended up choosing their Managed Prometheus solution but we faced the same problem and the solution was in fact to open up port 6443 in firewall, which is documented in their guide.

Maybe the Keda Troubleshooting page could drop a line about it.

https://keda.sh/docs/2.10/troubleshooting/