[Release-1.26] - Unable to access metrics-server pod ip from all nodes

[brandond]

Backport fix for Unable to access metrics-server pod ip from all nodes

• Unable to access metrics-server pod ip from all nodes #5518

[ShylajaDevadiga]

Validated fix on rke2 version v1.26.14-rc3+rke2r1

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release 
NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:
3 server 1 agent

Config.yaml:

ec2-user@ip-172-31-12-188:~> cat /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: "0644"
tls-san:
  - fake.fqdn.value
node-name: ip-172-31-12-188.us-east-2.compute.internal
profile: cis-1.23
cni: calico

Steps to validate

1. Install rke2 in HA setup
2. Check metrics on all nodes

ec2-user@ip-172-31-12-188:~> rke2 -v
rke2 version v1.26.14-rc3+rke2r1 (93518400c62043756be7e321480bcd5f19f42bbf)
go version go1.21.7 X:boringcrypto
ec2-user@ip-172-31-12-188:~> kubectl get pods -A -o wide|grep metrics
kube-system       helm-install-rke2-metrics-server-kjgd6                                 0/1     Completed   0            9h      10.42.87.197    ip-172-31-12-188.us-east-2.compute.internal   <none>           <none>
kube-system       rke2-metrics-server-84b8d89697-lbtht                                   1/1     Running     0            9h      10.42.87.201    ip-172-31-12-188.us-east-2.compute.internal   <none>           <none>
ec2-user@ip-172-31-12-188:~> kubectl top node
NAME                                          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-12-188.us-east-2.compute.internal   277m         13%    2299Mi          59%       
ip-172-31-12-196.us-east-2.compute.internal   276m         13%    1817Mi          46%       
ip-172-31-14-156.us-east-2.compute.internal   218m         10%    1620Mi          41%       
ip-172-31-9-54.us-east-2.compute.internal     42m          2%     766Mi           19%       
ec2-user@ip-172-31-12-188:~> 

Metrics is available in other nodes

ec2-user@ip-172-31-12-196:~> kubectl top node
NAME                                          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-12-188.us-east-2.compute.internal   293m         14%    2293Mi          59%       
ip-172-31-12-196.us-east-2.compute.internal   267m         13%    1844Mi          47%       
ip-172-31-14-156.us-east-2.compute.internal   213m         10%    1668Mi          43%       
ip-172-31-9-54.us-east-2.compute.internal     40m          2%     766Mi           19%       
ec2-user@ip-172-31-12-196:~> 

ec2-user@ip-172-31-14-156:~> kubectl top node
NAME                                          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-12-188.us-east-2.compute.internal   354m         17%    2337Mi          60%       
ip-172-31-12-196.us-east-2.compute.internal   293m         14%    1902Mi          49%       
ip-172-31-14-156.us-east-2.compute.internal   235m         11%    1677Mi          43%       
ip-172-31-9-54.us-east-2.compute.internal     77m          3%     765Mi           19%       
ec2-user@ip-172-31-14-156:~>