[Release-1.30] - Secondary etcd-only nodes do not reconnect to apiserver after outage if joined against an etcd-only node

[brandond]

Backport fix for Secondary etcd-only nodes do not reconnect to apiserver after outage if joined against an etcd-only node

• Secondary etcd-only nodes do not reconnect to apiserver after outage if joined against an etcd-only node #11311

[aganesh-suse]

Validated on release-1.30 branch with commit 51404d0

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent - 2 etcd, 1 cp, 1 agent node config.
or 
3 etcd, 2 cp, 1 agent configuration.

P.S - all nodes pointing to the main/first etcd server

Config.yaml:

etcd only node config.yaml:

token: xxxx
disable-apiserver: true
disable-controller-manager: true
disable-scheduler: true
node-taint:
- node-role.kubernetes.io/etcd:NoExecute
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
debug: true

Control plane only node config.yaml:

$ cat /etc/rancher/k3s/config.yaml 
token: xxxx
server: https://1.1.1.1:6443
disable-etcd: true
node-taint:
- node-role.kubernetes.io/control-plane:NoSchedule
write-kubeconfig-mode: "0644"
node-external-ip: 2.2.2.2
node-label:
- k3s-upgrade=server
debug: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='51404d0e83148bffa1a0cd8e9f8ef97fa7bb435e' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Restart control plane node. Wait for 5 minutes.
5. Verify cluster status - that all nodes are in Ready state:

kubectl get nodes -o wide

Replication Results:

• k3s version used for replication:

$ k3s -v 
k3s version v1.30.6+k3s1 (1829eaae)
go version go1.22.8

$ kubectl get nodes
DEBU[0000] Asset dir /var/lib/rancher/k3s/data/5aa71fa53ccd126a59cca0667c10675f13cae44186cd95944e782ee84022b62d
DEBU[0000] Running /var/lib/rancher/k3s/data/5aa71fa53ccd126a59cca0667c10675f13cae44186cd95944e782ee84022b62d/bin/kubectl [kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes]
NAME               STATUS     ROLES                  AGE   VERSION
ip-172-31-10-16    NotReady   <none>                 14m   v1.30.6+k3s1
ip-172-31-12-241   Ready      control-plane,master   16m   v1.30.6+k3s1
ip-172-31-6-146    NotReady   etcd                   16m   v1.30.6+k3s1
ip-172-31-7-101    Ready      etcd                   16m   v1.30.6+k3s1

Validation Results:

• k3s version used for validation:

$ k3s -v 
k3s version v1.30.6+k3s-51404d0e (51404d0e)
go version go1.22.8

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes 
DEBU[0000] Asset dir /var/lib/rancher/k3s/data/ab8d03ff8b000a0ca49963174ed2c810917415feab471cd31af2dce57e824654
DEBU[0000] Running /var/lib/rancher/k3s/data/ab8d03ff8b000a0ca49963174ed2c810917415feab471cd31af2dce57e824654/bin/kubectl [kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes]
NAME               STATUS   ROLES                  AGE   VERSION
ip-172-31-15-122   Ready    etcd                   56m   v1.30.6+k3s-51404d0e
ip-172-31-7-199    Ready    <none>                 54m   v1.30.6+k3s-51404d0e
ip-172-31-8-46     Ready    etcd                   56m   v1.30.6+k3s-51404d0e
ip-172-31-9-102    Ready    control-plane,master   56m   v1.30.6+k3s-51404d0e