cluster-reset stuck in infinite loop

[withinboredom]

Environmental Info:
K3s Version:

k3s version v1.26.6+k3s1 (3b1919b)
go version go1.19.10

Node(s) CPU architecture, OS, and Version:

Linux cantor 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 servers/agents (now 1? server)

Describe the bug:

Performed k3s server --cluster-reset, now stuck with the following loop during reset:

WARN[0111] Updating bootstrap data on disk from datastore
{"level":"warn","ts":"2023-07-01T22:14:32.037Z","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://65.108.75.198:2380/version","remote-member-id":"27b8752c1a5863fc","error":"Get \"https://65.108.75.198:2380/version\": dial tcp 65.108.75.198:2380: connect: connection refused"}
{"level":"warn","ts":"2023-07-01T22:14:32.037Z","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"27b8752c1a5863fc","error":"Get \"https://65.108.75.198:2380/version\": dial tcp 65.108.75.198:2380: connect: connection refused"}
{"level":"warn","ts":"2023-07-01T22:14:32.616Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"27b8752c1a5863fc","rtt":"0s","error":"dial tcp 65.108.75.198:2380: connect: connection refused"}
{"level":"warn","ts":"2023-07-01T22:14:32.616Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"27b8752c1a5863fc","rtt":"0s","error":"dial tcp 65.108.75.198:2380: connect: connection refused"}
INFO[0115] Defragmenting etcd database
{"level":"info","ts":"2023-07-01T22:14:34.050Z","caller":"v3rpc/maintenance.go:90","msg":"starting defragment"}
{"level":"info","ts":"2023-07-01T22:14:34.057Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","current-db-size-bytes":236642304,"current-db-size":"237 MB","current-db-size-in-use-bytes":236621824,"current-db-size-in-use":"237 MB"}
{"level":"info","ts":"2023-07-01T22:14:34.839Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","current-db-size-bytes-diff":4096,"current-db-size-bytes":236646400,"current-db-size":"237 MB","current-db-size-in-use-bytes-diff":4096,"current-db-size-in-use-bytes":236625920,"current-db-size-in-use":"237 MB","took":"789.063167ms"}
{"level":"info","ts":"2023-07-01T22:14:34.839Z","caller":"v3rpc/maintenance.go:96","msg":"finished defragment"}
{"level":"warn","ts":"2023-07-01T22:14:34.839Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-07-01T22:14:34.050Z","time spent":"789.179523ms","remote":"127.0.0.1:36064","response type":"/etcdserverpb.Maintenance/Defragment","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
INFO[0116] Reconciling bootstrap data between datastore and disk
INFO[0116] Cluster reset: backing up certificates directory to /var/lib/rancher/k3s/server/tls-1688249674
WARN[0116] Updating bootstrap data on disk from datastore

Steps To Reproduce:

Unknown... worked fine everytime previously.

Simply run: k3s server --cluster-reset and it will hang with the above logs.

Expected behavior:

To not be stuck in an infinite loop

Actual behavior:

Stuck in an infinite loop

Additional context / logs:

[brandond]

I've not seen this before. Can you attach the complete k3s service log, as well as the full output of the k3s server --cluster-reset command?

Also, what is 65.108.75.198 - is that this node, or another node? etcd shouldn't be attempting to connect to any other peers during reset, as it should have just been reset to a single-node cluster.

[withinboredom]

Sadly, the logs are lost to the sands of time. That IP would be itself, trying to connect to itself. I ran into this twice "experimenting" with a lost quorum/network split over dozens of iterations.

Part of this experiment was having etcd listening on multiple interfaces and forcing a network split on one of the routes. Etcd/k3s was able to resolve the fallback routes just fine. However, even after adding a --tls-san to get it to accept the connection, it appears etcd includes the ip/interface of the node in the etcd node name. This was a surprise because k8s doesn't allow multiple nodes with the exact same name, thus it should be a safe assumption that the actual ip address of the node shouldn't matter. However, this caused some nodes to see a quorum and others not -- depending on the route the packets took.

Dunno if that is a k3s or etcd thing, or if it is even a bug. But that's the context of these logs.

[brandond]

it appears etcd includes the ip/interface of the node in the etcd node name

etcd uses the node's primary IP address as its endpoint. Each node has an identifier and a fixed ip-based endpoint, and those must remain static through the node's life in the etcd cluster. If the address changes, the node needs to be deleted and re-added to the cluster, or the cluster needs to be reset to a single node with the current address, and the members re-added.

[withinboredom]

So there is no way to increase the reliability of etcd? Is this an etcd shortcoming then? If so, I can open an issue there.

[brandond]

This is how we have chosen to set up the cluster etcd endpoints. I don't think it's related to whatever you had going on with your cluster reset issue, but unfortunately without logs and/or a copy of the datastore it's very difficult to say what was going on.