Node cannot connect: lease is not expired (remaining ticks: 10)

[withinboredom]

Environmental Info:
K3s Version:

k3s version v1.27.3+k3s1 (fe9604c)
go version go1.20.5

Node(s) CPU architecture, OS, and Version:

Linux cameo 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 servers

Describe the bug:

Simply rebooted a node after draining. Other nodes show the following error:

Jul 19 00:31:36 cameo k3s[1579]: {"level":"info","ts":"2023-07-19T00:31:36.978Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e73fc2e93a0c43aa [logterm: 77, index: 51865323, vote: cac11d0841cdf5de] ignored MsgPreVote from 9933fe57c2365f4d [logterm: 76, index: 51861825] at term 77: lease is not expired (remaining ticks: 10)"}

The connecting node shows:

Jul 19 00:30:29 capital k3s[1504]: {"level":"info","ts":"2023-07-19T00:30:29.477Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9933fe57c2365f4d [logterm: 76, index: 51861825] sent MsgPreVote request to e73fc2e93a0c43aa at term 77"}

There are always 10 ticks remaining, whatever that means.

Steps To Reproduce:

• Unknown specifics. This was a simple drain and reboot.

Expected behavior:

Nodes to be able to rebooted and reconnect.

Actual behavior:

Node fails to rejoin cluster.

Additional context / logs:

This can be worked around by forcibly rebooting the leader node.

[brandond]

These are all info-level message from the embedded etcd instance regarding raft leader election; I don't see anything wrong based on just these messages. Can you attach the complete logs from all three nodes? Node reboots are very common, and I've not heard any reports of nodes failing to come back up after being rebooted, absent some other issue that causes data to be lost or discarded on reboot.

When you say that it "fails to rejoin the cluster", what exactly do you mean? It shouldn't be a matter of rejoining (thats term is usually reserved for a node that has been deleted and is joining again) - it should just be a matter of starting up already joined and replaying the raft log from another cluster member.

[withinboredom]

It eventually came back after awhile. It was just these log messages back and forth. K3s never came online on the rebooted node until it was resolved.

what exactly do you mean?

• The node was NotReady according to k3s.
• On the node, running kubectl get X resulted in an error due to k3s not fully 'booted' due to being unable to progress with etcd.

For giggles, I rebooted the node again and there wasn't an issue. So, maybe it was just some strange edge case.