How to remove orphaned master node details from etcd?

[belajarpowershell]

I have a HA cluster
1 Load Balancer
3 Master nodes
3 worker nodes.

Recently upgraded the cluster from "v1.25.6+k3s1" to "v1.26.10+k3s1".

During the upgrade , one master node had firewall issues and could not complete the upgrade.
I removed the node from the cluster, and the upgrade proceeded. (In hindsight I should not have done this)

After fixing the firewall issue, I installed the k3s as we would a new node.
But the k3s service did not start.

Here is what I have tried.

1. restarted k3s service on the failed master node.
2. uninstalled and reinstalled "v1.26.10+k3s1" on the failed master node.
3. tried kubectl delete node x9314.corp.aaa.net it says node note found.

journalctl -u k3s --since "3 minutes ago"
<snipped>
Apr 01 21:00:30 x9314.corp.aaa.net k3s[1091369]: time="2024-04-01T21:00:30+05:30" level=info msg="containerd is now running"
Apr 01 21:00:30 x9314.corp.aaa.net k3s[1091369]: time="2024-04-01T21:00:30+05:30" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-w>
Apr 01 21:00:30 x9314.corp.aaa.net k3s[1091369]: time="2024-04-01T21:00:30+05:30" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Apr 01 21:00:30 x9314.corp.aaa.net k3s[1091369]: time="2024-04-01T21:00:30+05:30" level=info msg="Handling backend connection request [x9314.corp.aaa.net]"
Apr 01 21:00:30 x9314.corp.aaa.net k3s[1091369]: time="2024-04-01T21:00:30+05:30" level=fatal msg="etcd cluster join failed: duplicate node name found, please use a unique name fo>
Apr 01 21:00:30 x9314.corp.aaa.net systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 21:00:30 x9314.corp.aaa.net systemd[1]: k3s.service: Failed with result 'exit-code'.
Apr 01 21:00:30 x9314.corp.aaa.net systemd[1]: Failed to start Lightweight Kubernetes.
Apr 01 21:00:36 x9314.corp.aaa.net systemd[1]: k3s.service: Scheduled restart job, restart counter is at 606.

Any ideas how I can rejoin the master x9314 node back to the cluster?

thank you .

Suresh

[brandond]

Do you still see the node in kubectl get node -o wide? Deleting the node from Kubernetes should also remove the node from the etcd cluster.

[belajarpowershell]

I was able to get the output with this.

sudo ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl  member list 

492dfc76470deea9, started, x9238.corp.aaa.net-afa9bacf, https://10.177.51.204:2380, https://10.177.51.204:2379
a69975be3788a446, started, x9314.corp.aaa.net-58569e8e, https://10.177.51.201:2380, https://10.177.51.201:2379
cf5f9964b8b17ae4, started, x9237.corp.aaa.net-8d1005d0, https://10.177.51.203:2380, https://10.177.51.203:2379

I will need to do member remove on the x9237 and x9238 , as there are only 2 master nodes I cannot risk loosing access. Any other gothcas I should look out for?

[brandond]

That's a slightly different way of doing what I suggested, but the end result is similar.

You're sure that you need to remove TWO nodes from the cluster? If two of your three nodes were offline the cluster would lack quorum and be non-functional.

[belajarpowershell]

I was not clear .
To remove (a69975be3788a446, started, x9314.corp.aaa.net-58569e8e) , do I need to run the command

etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt member remove a69975be3788a446

on both nodes x9238 and x9237?

x9238 and x9237 will remain online.

And if I run the member remove command do I need to take any precautions?

thank you

[brandond]

no, you only need to remove it from the cluster once.

[belajarpowershell]

Thank you , this worked for me. the x9314 has rejoined the cluster .