Unable to join 3rd server node to HA Embedded Etcd

[Crowley723]

I have two k3s server nodes running on Ubuntu-Server Proxmox VMs. (192.168.10.200, and 192.168.10.201) I am attempting to enroll the third node which will be running on my Bare metal Rocky Linux 9 Host. When I attempt to join it using this command

curl -sfL https://get.k3s.io | K3S_TOKEN=MY_TOKEN sh -s - server     
--server https://192.168.10.200:6443     
--tls-san=192.168.10.26     
--bind-address 192.168.10.26     
--node-ip 192.168.10.26     
--disable-cloud-controller 
--selinux

I get this error:

Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=error msg="Sending HTTP 503 response to 192.168.10.12:59810: runtime core not ready"

I think this may be related to the fact that I am attempting to run k3s on a different IP (192.168.10.26) but the same NIC as my regular docker services on this host (192.168.10.12).
I have tried to join this server before and may have used the wrong tags for the command which resulted in some of the configs grabbing the wrong IP address.

I have tried uninstalling k3s and reinstalling but that doesn't seem to completely remove the errant config file with the wrong IP.

Any suggestions for what I can try to get this resolved would be greatly appreciated.

My Network settings

2: enp4s0:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 04:42:1a:**:**:f6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.12/24 brd 192.168.10.255 scope global noprefixroute enp4s0
       valid_lft forever preferred_lft forever
    inet 192.168.10.26/24 brd 192.168.10.255 scope global secondary noprefixroute enp4s0
       valid_lft forever preferred_lft forever
    inet6 2600:1700:****:****::****/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::642:****:****:****/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

---

Here are the full logs

Nov 23 18:42:51 Moor k3s[2163907]: time="2024-11-23T18:42:51-08:00" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
Nov 23 18:42:51 Moor k3s[2163907]: time="2024-11-23T18:42:51-08:00" level=info msg="To join server node to cluster: k3s server -s https://192.168.10.26:6443 -t ${SERVER_NODE_TOKEN}"
Nov 23 18:42:51 Moor k3s[2163907]: time="2024-11-23T18:42:51-08:00" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
Nov 23 18:42:51 Moor k3s[2163907]: time="2024-11-23T18:42:51-08:00" level=info msg="To join agent node to cluster: k3s agent -s https://192.168.10.26:6443 -t ${AGENT_NODE_TOKEN}"
Nov 23 18:42:51 Moor k3s[2163907]: time="2024-11-23T18:42:51-08:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Nov 23 18:42:51 Moor k3s[2163907]: time="2024-11-23T18:42:51-08:00" level=info msg="Run: k3s kubectl"
Nov 23 18:42:51 Moor k3s[2163907]: time="2024-11-23T18:42:51-08:00" level=info msg="Password verified locally for node moor"
Nov 23 18:42:51 Moor k3s[2163907]: time="2024-11-23T18:42:51-08:00" level=info msg="certificate CN=moor signed by CN=k3s-server-ca@1732409154: notBefore=2024-11-24 00:45:54 +0000 UTC notAfte
r=2025-11-24 02:42:51 +0000 UTC"
Nov 23 18:42:52 Moor k3s[2163907]: time="2024-11-23T18:42:52-08:00" level=info msg="certificate CN=system:node:moor,O=system:nodes signed by CN=k3s-client-ca@1732409154: notBefore=2024-11-24
 00:45:54 +0000 UTC notAfter=2025-11-24 02:42:52 +0000 UTC"
Nov 23 18:42:52 Moor k3s[2163907]: time="2024-11-23T18:42:52-08:00" level=info msg="Module overlay was already loaded"
Nov 23 18:42:52 Moor k3s[2163907]: time="2024-11-23T18:42:52-08:00" level=info msg="Module nf_conntrack was already loaded"
Nov 23 18:42:52 Moor k3s[2163907]: time="2024-11-23T18:42:52-08:00" level=info msg="Module br_netfilter was already loaded"
Nov 23 18:42:52 Moor k3s[2163907]: time="2024-11-23T18:42:52-08:00" level=info msg="Module iptable_filter was already loaded"
Nov 23 18:42:52 Moor k3s[2163907]: time="2024-11-23T18:42:52-08:00" level=info msg="Found nvidia container runtime at /usr/bin/nvidia-container-runtime"
Nov 23 18:42:52 Moor k3s[2163907]: time="2024-11-23T18:42:52-08:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Nov 23 18:42:52 Moor k3s[2163907]: time="2024-11-23T18:42:52-08:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=info msg="containerd is now running"
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=info msg="Connecting to proxy" url="wss://192.168.10.26:6443/v1-k3s/connect"
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=info msg="Creating k3s-cert-monitor event broadcaster"
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=info msg="Running kubelet --address=192.168.10.26 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=moor --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-ip=192.168.10.26 --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=info msg="Handling backend connection request [moor]"
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=info msg="Remotedialer connected to proxy" url="wss://192.168.10.26:6443/v1-k3s/connect"
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=info msg="Starting etcd to join cluster with members [k3s-master-1-63dd7399=https://192.168.10.200:2380 moor-b92bdbf5=https://192.168.10.26:2380 k3s-master-2-5c722291=https://192.168.10.201:2380]"
Nov 23 18:42:53 Moor k3s[2163907]: {"level":"info","ts":"2024-11-23T18:42:53.288253-0800","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["https://127.0.0.1:2380","https://192.168.10.26:2380"]}
Nov 23 18:42:53 Moor k3s[2163907]: {"level":"info","ts":"2024-11-23T18:42:53.288329-0800","caller":"embed/etcd.go:494","msg":"starting with peer TLS","tls-info":"cert = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt, key = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key, client-cert=, client-key=, trusted-ca = /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
Nov 23 18:42:53 Moor k3s[2163907]: {"level":"info","ts":"2024-11-23T18:42:53.288682-0800","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://192.168.10.26:2379"]}
Nov 23 18:42:53 Moor k3s[2163907]: {"level":"info","ts":"2024-11-23T18:42:53.288763-0800","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.13","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.22.8","go-os":"linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":false,"name":"moor-b92bdbf5","data-dir":"/var/lib/rancher/k3s/server/db/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/k3s/server/db/etcd/member","force-new-cluster":false,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["https://127.0.0.1:2380","https://192.168.10.26:2380"],"advertise-client-urls":["https://192.168.10.26:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.10.26:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"k3s-master-1-63dd7399=https://192.168.10.200:2380,k3s-master-2-5c722291=https://192.168.10.201:2380,moor-b92bdbf5=https://192.168.10.26:2380","initial-cluster-state":"existing","initial-cluster-token":"etcd-cluster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
Nov 23 18:42:53 Moor k3s[2163907]: {"level":"info","ts":"2024-11-23T18:42:53.29028-0800","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","took":"1.29818ms"}
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=error msg="Sending HTTP 503 response to 192.168.10.12:59810: runtime core not ready"
Nov 23 18:42:53 Moor k3s[2163907]: time="2024-11-23T18:42:53-08:00" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=moor --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Nov 23 18:42:53 Moor k3s[2163907]: E1123 18:42:53.388692 2163907 server.go:1051] "Failed to retrieve node info" err="apiserver not ready"
Nov 23 18:42:54 Moor k3s[2163907]: E1123 18:42:54.469001 2163907 server.go:1051] "Failed to retrieve node info" err="apiserver not ready"
Nov 23 18:42:56 Moor k3s[2163907]: E1123 18:42:56.574028 2163907 server.go:1051] "Failed to retrieve node info" err="apiserver not ready"