k3s server behavior in a partitioned network scenario

[Dadbod88]

I have a 5 node cluster each running in server mode. Each node has 4 ethernet ports, and connected to two leaf switches via two interfaces. So nodes can communicate to other nodes via one of the leaf switches. Now if we disconnect two cables on two nodes such that they are connected to different leaf switch we basically disconnect two nodes from each other. Say on node-1, we take out two cables via leaf-1 leaving node-1 only connected to leaf-2, similarly on node-5 we take out cables via leaf-2 leaving node-5 only connected to leaf-1. Now remaining nodes are not configured to act as a router for node-1 to node-5, node-1 and node-5 are disconnected. So for node-1 the cluster is running with 4 nodes, node 1 to 4, and for node-5 cluster has 4 nodes, node 2 to 5. For node 2 to 4 all nodes are connected and reachable. Now in this situation it is possible for some of the lease heartbeat missing if api server is on node-1 and kubelet on node-5 and vice versa. I don't have any external load balancer and kubernetes api service is DNATed to nodes k3s server with random probability. Is this condition kubernetes should be handle on its own ? Occasionally I end up with unhealthy cluster as reported in etcdctl endpoint health. And some of the deployment with replica count 1, they get scheduled on other nodes while the pod running on cabled out node is still active and doesn't receive SIGTERM/SIGKILL.

[brandond]

I'm not really following your setup here. I will note that the apiservers do not communicate directly between each other; everything is handled through leases in etcd. The question you should be trying to answer is whether or not your etcd cluster still has quorum in any given scenario. Are 3 out of 5 of your server nodes able to reach each other?

[Dadbod88]

3 out of 5 server nodes are able to communicate to each other. Two nodes are completely disconnected and I have the following

`etcdctl -w table endpoint health

{"level":"warn","ts":"2024-03-04T17:36:43.922Z","logger":"client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000408a80/172.16.2.5:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
+-------------------------+--------+--------------+---------------------------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+-------------------------+--------+--------------+---------------------------+
| https://172.16.2.4:2379 | true | 3.21167ms | |
| https://172.16.2.2:2379 | true | 3.119791ms | |
| https://172.16.2.1:2379 | true | 3.538574ms | |
| https://172.16.2.3:2379 | true | 3.542945ms | |
| https://172.16.2.5:2379 | false | 5.000177974s | context deadline exceeded |
+-------------------------+--------+--------------+---------------------------+
Error: unhealthy cluster`

The above is captured on a node whose connections are intact. In this case the node-5 (172.16.2.5) and node-1(172.16.2.1) were disconnected from each other.

[Dadbod88]

We have a deployment with replica count 1 which runs behind a service.. In above case that pod were running on node-5. Occasionally we have that pod is marked "Terminating", however I checked the containerd logs on that node (node-5 in above case) that no SIGKILL/SIGTERM is sent. So the process inside the pod is alive. When I dumped the endpointslice it contains both the old pod and new pod, however the old pod has .endpoints[0].conditions.[ready: false, serving: false, terminating: true] which is correct as api server knows the pod is in terminating. However the old pod not only running, but it is serving requests as other pods in the cluster are reaching through it via service IP and the flows are cached in conntrack table. In this situation is there a way to ensure the new pod is scheduled only if the old instance is terminated (i.e the process is terminated) as etcd is not healthy can we rely on kubernetes to ensure this ?