[Release-1.28] - Duplicate endpoints prevents any endpoint from being used

[brandond]

Backport fix for Duplicate endpoints prevents any endpoint from being used

• Duplicate endpoints prevents any endpoint from being used #5577

[aganesh-suse]

Validated on release-1.28 branch with commit 8fabb98

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA : 3 server / 1 agent

or

1 server/ 1 agent

Config.yaml:

token: xxxx
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
debug: true

Additional files

registries.yaml

mirrors:
  docker.io:
    endpoint:
      - https://registry.example.com
      - https://registry.example.com 

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

2. Install RKE2

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT='8fabb98a17bddf06fb387ff274e7b3af90819ad8' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -

3. Start the RKE2 service

$ sudo systemctl enable --now rke2-server
or 
$ sudo systemctl enable --now rke2-agent

4. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

5. Check hosts.toml contents:

sudo cat /var/lib/rancher/rke2/agent/etc/containerd/certs.d/docker.io/hosts.toml

Check containerd logs for errors:

sudo cat /var/lib/rancher/rke2/agent/containerd/containerd.log | grep error | grep TOML on server1

Replication Results:

• rke2 version used for replication:

$ rke2 -v
rke2 version v1.28.7+rke2r1 (fe8538a165d25755050d2a0827dbf982f72611fa)
go version go1.21.7 X:boringcrypto

$ sudo cat /var/lib/rancher/rke2/agent/etc/containerd/certs.d/docker.io/hosts.toml
# File generated by rke2. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"

[host."https://registry.example.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

[host."https://registry.example.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

$ sudo cat /var/lib/rancher/rke2/agent/containerd/containerd.log | grep error | grep TOML on server1
time="2024-03-15T02:15:17.509832797Z" level=error msg="failed to decode hosts.toml" error="failed to parse TOML: (8, 2): duplicated tables"
time="2024-03-15T02:15:17.660226037Z" level=error msg="failed to decode hosts.toml" error="failed to parse TOML: (8, 2): duplicated tables"

Validation Results:

• rke2 version used for validation:

$ rke2 -v
rke2 version v1.28.7+dev.8fabb98a (8fabb98a17bddf06fb387ff274e7b3af90819ad8)
go version go1.21.7 X:boringcrypto

$ sudo cat /var/lib/rancher/rke2/agent/etc/containerd/certs.d/docker.io/hosts.toml
# File generated by rke2. DO NOT EDIT.

server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]



[host."https://registry.example.com/v2"]
  capabilities = ["pull", "resolve"]

$ sudo cat /var/lib/rancher/rke2/agent/containerd/containerd.log | grep error | grep TOML on server1