Unable to access metrics-server pod ip from all nodes

[ShylajaDevadiga]

Environmental Info:
RKE2 Version:
v1.29.2-rc2+rke2r1
v1.28.7-rc2+rke2r1

Node(s) CPU architecture, OS, and Version:

NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:
3 server nodes
cni: calico
profile: cis

Describe the bug:

1. Failure to access metrics-server pod ip from other nodes than the node that the pod was deployed
   Logs from the api-server pod

E0223 18:12:08.012410       1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable

2. namespace deletion stuck in Terminating with finalizers

spec:
  finalizers:
  - kubernetes

Steps To Reproduce:

1. Install rke2 on multinode cluster using cni calico on cis enabled mode
2. Check apiserver logs
3. Run kubectl top node on all nodes

Expected behavior:
Should be able to access metrics from any node

> kubectl top node
NAME                                        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-1-58.us-east-2.compute.internal   254m         12%    2166Mi          55%       
ip-172-31-4-46.us-east-2.compute.internal   304m         15%    1762Mi          45%       
ip-172-31-6-72.us-east-2.compute.internal    243m         12%    1650Mi          42%      

Actual behavior:
From other nodes

kubectl top node
error: Metrics API not available

> curl -vks https://10.42.252.200:10250/apis/metrics.k8s.io/v1beta1
*   Trying 10.42.252.200:10250...
^C

[brandond]

@andreufontb the change responsible for this regression is not present in any GA release. Please open a new issue.

[ShylajaDevadiga]

Validated fix on rke2 version v1.29.2-rc3+rke2r1

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release 
ec2-user@ip-172-31-4-114:~> cat /etc/os-release
NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:
3 server 1 agent

Config.yaml:

cat /etc/rancher/rke2/config,yaml
write-kubeconfig-mode: "0644"
tls-san:
  - fake.fqdn.value
node-name: ip-172-31-4-114.us-east-2.compute.internal
profile: cis-1.23
cni: calico

Steps to validate

1. Install rke2 in HA setup
2. Check metrics

ec2-user@ip-172-31-4-114:~> rke2 -v
rke2 version v1.29.2-rc3+rke2r1 (08699dfffdf75a61a5e6064f9f8efe8ddae857fe)
go version go1.21.7 X:boringcrypto
ec2-user@ip-172-31-4-114:~> kubectl get pods -A -o wide|grep metrics
kube-system       helm-install-rke2-metrics-server-jmwf2                                0/1     Completed   0               4h1m    10.42.81.199    ip-172-31-4-114.us-east-2.compute.internal   <none>           <none>
kube-system       rke2-metrics-server-544c8c66fc-jw2mh                                  1/1     Running     0               3h59m   10.42.81.201    ip-172-31-4-114.us-east-2.compute.internal   <none>           <none>
ec2-user@ip-172-31-4-114:~> kubectl top node
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-2-71.us-east-2.compute.internal    269m         13%    1536Mi          39%       
ip-172-31-4-114.us-east-2.compute.internal   243m         12%    2247Mi          58%       
ip-172-31-7-107.us-east-2.compute.internal   280m         14%    1518Mi          39%       
ip-172-31-9-171.us-east-2.compute.internal   48m          2%     710Mi           18%       
ec2-user@ip-172-31-4-114:~> 

Metrics is available in other nodes

ec2-user@ip-172-31-2-71:~> kubectl top node
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-2-71.us-east-2.compute.internal    241m         12%    1549Mi          40%       
ip-172-31-4-114.us-east-2.compute.internal   277m         13%    2249Mi          58%       
ip-172-31-7-107.us-east-2.compute.internal   224m         11%    1517Mi          39%       
ip-172-31-9-171.us-east-2.compute.internal   51m          2%     710Mi           18%       
ec2-user@ip-172-31-2-71:~> 

ec2-user@ip-172-31-7-107:~> kubectl top node
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-2-71.us-east-2.compute.internal    529m         26%    1608Mi          41%       
ip-172-31-4-114.us-east-2.compute.internal   447m         22%    2300Mi          59%       
ip-172-31-7-107.us-east-2.compute.internal   557m         27%    1559Mi          40%       
ip-172-31-9-171.us-east-2.compute.internal   300m         15%    717Mi           18%       
ec2-user@ip-172-31-7-107:~>