Omni erroneously removes a live etcd member

[smira]

#745

[justyns]

FWIW, I also ran into this issue when upgrading from Talos 1.8.1 to 1.8.4. I didn't do anything special, just updated the cluster template and ran omnictl cluster template sync. The first node upgraded without issue, the second control plane node got stuck in an etcd restart loop with errors like:

100.64.0.77: {"level":"info","ts":"2024-12-24T22:14:15.262169Z","caller":"etcdserver/server.go:864","msg":"starting etcd server","local-member-id":"66de3babc0abe198","local-server-version":"3.5.17","cluster-id":"78fa62a055012012","cluster-version":"3.5"}                           
100.64.0.77: {"level":"info","ts":"2024-12-24T22:14:15.262330Z","caller":"etcdserver/server.go:773","msg":"starting initial election tick advance","election-ticks":5}                                                                                                                   
100.64.0.77: {"level":"warn","ts":"2024-12-24T22:14:15.262417Z","caller":"etcdserver/server.go:1154","msg":"server error","error":"the member has been permanently removed from the cluster"}                                                                                            
100.64.0.77: {"level":"warn","ts":"2024-12-24T22:14:15.262494Z","caller":"etcdserver/server.go:1155","msg":"data-dir used by this member must be removed"}                                                                                                                               

To "fix" it, I rebooted and selected the "Reset installation and reboot into maintenance mode" option for the 100.64.0.77 node. It eventually came back up.

I made a support bundle with omnictl before resetting the node. Would it be helpful to send somewhere?

[Unix4ever]

Did you have any extensions installed on the nodes?

[Unix4ever]

I've identified the root cause. The fix has landed in 0.45.1.