Cant join the 3rd master to new cluster

[ksl28]

Environmental Info:
RKE2 Version:

rke2 version v1.26.10+rke2r2 (21e3a8c)
go version go1.20.10 X:boringcrypto

Node(s) CPU architecture, OS, and Version:

Linux dk1k8s01 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

The cluster consists of 2 etcd nodes, and i want to add a third one, before i add the agent nodes.
dk1k8sclu01 / 192.168.20.50 - vip / keepalived
dk1k8s01 / 192.168.20.51 - working (current holder of the Keepalived VIP)
dk1k8s02 / 192.168.20.52 - working
dk1k8s07 / 192.168.20.53 - faulty

root@dk1k8s01:~# /var/lib/rancher/rke2/bin/kubectl get nodes --kubeconfig /etc/rancher/rke2/rke2.yaml
NAME STATUS ROLES AGE VERSION
dk1k8s01 Ready control-plane,etcd,master 2d22h v1.26.10+rke2r2
dk1k8s02 Ready control-plane,etcd,master 2d21h v1.26.10+rke2r2

Describe the bug:
So i created 3 identical Ubuntu 22.04 VMs on Proxmox, and follow the same simple guide from Rancher, on how to install the platform. The first 2 nodes worked as expected, but the third one always fails to join the cluster.
I have performed the following actions:

1. Completely erased the 3rd node, and installed it from scratch with a new name - but same IP.
2. removed everything under /var/lib/rancher/rke2 on the faulty node
3. disabled ufw / apparmor (https://www.reddit.com/r/rancher/comments/17va99x/comment/k9p35ku/?utm_source=share&utm_medium=web2x&context=3)
4. Ensured that the nodes can reach each other on 9345

When i join the 3rd node to the cluster i can see that it starts consuming a lot of CPU and memory, and after about 5 minutes it fails.
The log on the 3rd node just get flooded with this type of messages:

Nov 23 18:21:39 dk1k8s07 rke2[4679]: time="2023-11-23T18:21:39Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Nov 23 18:21:39 dk1k8s07 rke2[4679]: {"level":"warn","ts":"2023-11-23T18:21:39.847636Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000969340/192.168.20.52:2379","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}

Additional context / logs:

faulty_node.txt

Im a bit lost here, and any help would be greatly appreciated! :)

[brandond]

Check the etcd pod logs under /var/log/pods. If you can't find anything interesting in there, attach all the pod logs, and the complete rke2-server logs from journald on the 3rd node.

[ksl28]

Check the etcd pod logs under /var/log/pods. If you can't find anything interesting in there, attach all the pod logs, and the complete rke2-server logs from journald on the 3rd node.

I have made some progress, and discovered that the 3rd node is not listening on TCP:2380 - but i havent had any luck, determining why its not doing that.

faulty_node_journal_log.txt
faulty-etcd-node-pods.tar.gz
working-etcd-node-pods.tar.gz

Update:
It seems that the issue is due to missing network config (CNI / CRI), so there are no pods running on the faulty node. - Containerd log:

time="2023-11-25T20:59:23.319477425Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"

I checked the folder /etc/cni/net.d on the faulty node, and its completely empty - unlike the other working nodes.
faulty-node-containerd.log
faulty-node-kubelet.log

Can i somehow generate the config files?

[brandond]

Did you check the etcd pod logs as suggested? It's not listening on the etcd port, and there are errors about connecting to etcd... but it sounds like you haven't checked the actual etcd logs yet.

[ksl28]

Did you check the etcd pod logs as suggested? It's not listening on the etcd port, and there are errors about connecting to etcd... but it sounds like you haven't checked the actual etcd logs yet.

Sorry for leaving that part out - so i wanted to check the etcd logs under /var/logs/pods, but the folder is completely empty:

root@dk1k8s07:/# ls -lt /var/log/pods/
total 0

So i searched for that and found this post - and believed i had the same issue.
#2080

I am new to K8S, so i might have misunderstood question - if so, please let me know :)

[brandond]

From the journald log:

Nov 25 19:36:37 dk1k8s07 rke2[1584]: time="2023-11-25T19:36:37Z" level=info msg="Waiting for other members to finish joining etcd cluster: etcdserver: unhealthy cluster"
Nov 25 19:36:38 dk1k8s07 rke2[1584]: time="2023-11-25T19:36:38Z" level=info msg="Adding member dk1k8s07-7c237374=https://192.168.20.53:2380 to etcd cluster [dk1k8s03-10304e89=https://192.168.20.53:2380 dk1k8s02-6bf5ea03=https://192.168.20.52:2380 dk1k8s01-a08f7f75=https://192.168.20.51:2380]"
Nov 25 19:36:38 dk1k8s07 rke2[1584]: {"level":"warn","ts":"2023-11-25T19:36:38.037737Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000cfe700/192.168.20.52:2379","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}

The 3rd node can't join because the 2nd node hasn't finished joining yet. Make sure that all of the correct ports are open between your nodes - see https://docs.rke2.io/install/requirements#inbound-network-rules for the list, taking particular note of the etcd ports. The etcd pod logs on the 1st and 2nd nodes probably have more information about the state of the cluster.

edit:
it actually looks like this node is already in the cluster, but with a different name.
Adding member dk1k8s07-7c237374=https://192.168.20.53:2380 to etcd cluster [dk1k8s03-10304e89=https://192.168.20.53:2380 dk1k8s02-6bf5ea03=https://192.168.20.52:2380 dk1k8s01-a08f7f75=https://192.168.20.51:2380

Was this node (dk1k8s07) previously a member of the cluster, under the name dk1k8s03? Or is this new node just using the same IP? Either way, I suspect you will need to delete that node from the cluster (using kubectl delete node) prior to joining this one.

You can't join a 4th node to a 3-node cluster while one of them is unhealthy.

[ksl28]

From the journald log:

Nov 25 19:36:37 dk1k8s07 rke2[1584]: time="2023-11-25T19:36:37Z" level=info msg="Waiting for other members to finish joining etcd cluster: etcdserver: unhealthy cluster"
Nov 25 19:36:38 dk1k8s07 rke2[1584]: time="2023-11-25T19:36:38Z" level=info msg="Adding member dk1k8s07-7c237374=https://192.168.20.53:2380 to etcd cluster [dk1k8s03-10304e89=https://192.168.20.53:2380 dk1k8s02-6bf5ea03=https://192.168.20.52:2380 dk1k8s01-a08f7f75=https://192.168.20.51:2380]"
Nov 25 19:36:38 dk1k8s07 rke2[1584]: {"level":"warn","ts":"2023-11-25T19:36:38.037737Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000cfe700/192.168.20.52:2379","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}

The 3rd node can't join because the 2nd node hasn't finished joining yet. Make sure that all of the correct ports are open between your nodes - see https://docs.rke2.io/install/requirements#inbound-network-rules for the list, taking particular note of the etcd ports. The etcd pod logs on the 1st and 2nd nodes probably have more information about the state of the cluster.

edit: it actually looks like this node is already in the cluster, but with a different name. Adding member dk1k8s07-7c237374=https://192.168.20.53:2380 to etcd cluster [dk1k8s03-10304e89=https://192.168.20.53:2380 dk1k8s02-6bf5ea03=https://192.168.20.52:2380 dk1k8s01-a08f7f75=https://192.168.20.51:2380

Was this node (dk1k8s07) previously a member of the cluster, under the name dk1k8s03? Or is this new node just using the same IP? Either way, I suspect you will need to delete that node from the cluster (using kubectl delete node) prior to joining this one.

You can't join a 4th node to a 3-node cluster while one of them is unhealthy.

Hi - thanks for the reply.

dk1k8s07 are not present in the kubectl get nodes:

dk1k8s03 was originally thought the be the 3rd master, but i never succeeded in getting in the cluster - so i reinstalled it with a new name (dk1k8s07), but the same IP address (192.168.20.53).

UFW and AppArmor are disabled on all the nodes.

I really appreciate your time on this! Let me know what to do now :)

[brandond]

Hmm. It would be exceedingly difficult for the node to be present in etcd without also joining the Kubernetes cluster, but it sounds like you've got some pretty screwy things going on with this cluster.

You could use etcdctl to remove the stale node, but easier than that would probably be to stop rke2 on all nodes, run rke2 server --cluster-reset on the 1st, remove the DB directory from the 2nd and 3rd nodes, and then start all three up one at a time.

[ksl28]

Hmm. It would be exceedingly difficult for the node to be present in etcd without also joining the Kubernetes cluster, but it sounds like you've got some pretty screwy things going on with this cluster.

You could use etcdctl to remove the stale node, but easier than that would probably be to stop rke2 on all nodes, run rke2 server --cluster-reset on the 1st, remove the DB directory from the 2nd and 3rd nodes, and then start all three up one at a time.

And just to be sure, it means i should delete everything under "/var/lib/rancher" on the 2nd and 3rd node - correct? :)

[brandond]

No, just the server/db directory. The cluster-reset will confirm that for you when it completes.

[ksl28]

No, just the server/db directory. The cluster-reset will confirm that for you when it completes.

I performed these steps:

1. Ran systemctl stop rke2-server.service on all nodes
2. Removed the db folder on dk1k8s02 & dk1k8s07 - rm -rf /var/lib/rancher/rke2/server/db
3. Confirmed that the service was stopped and the the db folder were no longer present on the secondary nodes.

Then i tried to run the rke2 server --cluster-reset on dk1k8s02, but keep getting this error message:

FATA[0020] cannot perform cluster-reset while server URL is set - remove server from configuration before resetting

Update:
So i found this post, regarding the --server flag - #3178
So i tried to remove the config.yaml from the first node, and then ran rke2 server --cluster-reset - this time it executed, but got stuck in a loop for about 15 minutes with this error:

INFO[0930] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error
{"level":"warn","ts":"2023-11-27T17:11:48.198932Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007361c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}
{"level":"info","ts":"2023-11-27T17:11:48.198982Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
{"level":"warn","ts":"2023-11-27T17:11:48.375182Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000948c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}
WARN[0934] Failed to get apiserver address from etcd: context deadline exceeded
INFO[0935] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error
INFO[0939] Pod for etcd not synced (pod sandbox not found), retrying
{"level":"warn","ts":"2023-11-27T17:11:53.375404Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000948a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}
WARN[0939] Failed to get apiserver address from etcd: context deadline exceeded
INFO[0940] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error
FATA[0942] failed to wait for apiserver ready: timed out waiting for the condition, failed to get apiserver /readyz status: Get "https://127.0.0.1:6443/readyz": read tcp 127.0.0.1:42586->127.0.0.1:6443: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:42554->127.0.0.1:6443: read: connection reset by peer
root@dk1k8s01:~#

Any suggestions?

[ksl28]

So i decided to spawn 3 completely new nodes on 3 new IP addresses, and followed the same steps as previously - but this time it worked.
So i finally have 3 master nodes in my cluster.

Thanks SO much, for the time debugging on this :)