[Release-1.27] - Unable to access metrics-server pod ip from all nodes

[brandond]

Backport fix for Unable to access metrics-server pod ip from all nodes

• Unable to access metrics-server pod ip from all nodes #5518

[ShylajaDevadiga]

Validated fix on rke2 version v1.27.11-rc3+rke2r1

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release 
> cat /etc/os-release
NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:
3 server 1 agent

Config.yaml:

cat /etc/rancher/rke2/config,yaml
write-kubeconfig-mode: "0644"
tls-san:
  - fake.fqdn.value
profile: cis-1.23
cni: calico

Steps to validate

1. Install rke2 in HA setup
2. Check metrics on all nodes

**Validation results: **

ec2-user@ip-172-31-4-100:~> rke2 -v
rke2 version v1.27.11-rc3+rke2r1 (6665618680112568f79b1f5992aecf4655e3cf8b)
go version go1.21.7 X:boringcrypto
ec2-user@ip-172-31-4-100:~> kubectl get pods -A -o wide|grep metrics
kube-system       helm-install-rke2-metrics-server-8bqx7                                0/1     Completed   0               7h40m   10.42.100.67    ip-172-31-4-100.us-east-2.compute.internal   <none>           <none>
kube-system       rke2-metrics-server-5c9768ff67-tkczm                                  1/1     Running     0               7h39m   10.42.100.73    ip-172-31-4-100.us-east-2.compute.internal   <none>           <none>
ec2-user@ip-172-31-4-100:~> kubectl top nodes
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-0-173.us-east-2.compute.internal   255m         12%    1655Mi          42%       
ip-172-31-3-205.us-east-2.compute.internal   274m         13%    1788Mi          46%       
ip-172-31-4-100.us-east-2.compute.internal   302m         15%    2361Mi          61%       
ip-172-31-8-102.us-east-2.compute.internal   40m          2%     754Mi           19%       
ec2-user@ip-172-31-4-100:~>

ec2-user@ip-172-31-0-173:~> kubectl top nodes
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-0-173.us-east-2.compute.internal   289m         14%    1654Mi          42%       
ip-172-31-3-205.us-east-2.compute.internal   333m         16%    1788Mi          46%       
ip-172-31-4-100.us-east-2.compute.internal   298m         14%    2360Mi          61%       
ip-172-31-8-102.us-east-2.compute.internal   69m          3%     753Mi           19%       
ec2-user@ip-172-31-0-173:~> 

ec2-user@ip-172-31-3-205:~> kubectl top nodes
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-0-173.us-east-2.compute.internal   236m         11%    1655Mi          42%       
ip-172-31-3-205.us-east-2.compute.internal   255m         12%    1800Mi          46%       
ip-172-31-4-100.us-east-2.compute.internal   350m         17%    2338Mi          60%       
ip-172-31-8-102.us-east-2.compute.internal   49m          2%     755Mi           19%       
ec2-user@ip-172-31-3-205:~>