Does k3s with ipv6 services require host ipv6 addresses to be static?

[kelvie]

One of my ipv6 services (I'm using metallb on some raspberry pis) stopped responding -- when I took a look at the ip6tables output, it looks like some of the rules still point to an old ipv6 address -- e.g.:

-A KUBE-SEP-GSBUBYA3I2W5J7GE -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination [fd08:9428:579a:0:d82d:3bf3:78ec:f9d8]:53

How does k3s choose which ipv6 address (there are many) from an interface is chosen for a service? I haven't used the fd08:: prefix address in a while -- does k3s just choose a random ipv6 address that happens to be on the node's interface during install, assuming it's static throughout the node's lifetime?

Is the best practice then to set up static ipv6 addresses for k3s hosts?

[brandond]

If you're using metallb to expose the service as a LoadBalancer, metallb is responsible for assigning the IP from one of its pools. The ClusterIP for the service is assigned once when the Service is created, out of the pool specified by the service-cidr configuration value.

[kelvie]

Sure, and that's all working fine. The advertised address is correct, but it gets lost because the load balancing appears to actually be done by ip6tables, and uses (I think) the node's internalIP as the final destination

-A KUBE-SERVICES -d fd00::5353/128 -p udp -m comment --comment "kube-system/kube-dns:dns loadbalancer IP" -m udp --dport 53 -j KUBE-EXT-TCOU7JCQXEZGVUNU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> [fd08:9428:579a:0:d82d:3bf3:78ec:f9d8]:53" -j KUBE-SEP-GSBUBYA3I2W5J7GE
-A KUBE-SEP-GSBUBYA3I2W5J7GE -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination [fd08:9428:579a:0:d82d:3bf3:78ec:f9d8]:53

If I had two pods, I think this changes to randomly loadbalance between two chains, one for each pod.

In either case, the final rule here uses an ipv6 address that no longer exists -- ipv6 addresses come and go -- but presumably existed at the time when I installed k3s on here.

I'm just curious as to how this is chosen. I did a bit more digging, and it seems like kube-proxy is in charge of creating these iptables rules -- where does it get this IP from? How was it chosen in the first place if an interface had multiple ipv6 addresses? Is it stored in etcd somewhere? (I see a reference to it in one of the certs for this node)

[brandond]

Yes, the node IPs (of both IP families) are expected to be static. Kubernetes does not handle node IP changing while it is running, as you would see if you are using reservationless dynamic addressing for your nodes.

[kelvie]

Every single node IP needs to be static? So the best practice here is to make sure that interfaces have static IPv6 addresses for every prefix they are a part of (and don't take on more via router advertisements) ?

Or is there a way to let kube-proxy (or kubelet?) know which ipv6 address to use?

It seems to just choose one, and I'm not sure how it chooses.

[brandond]

Starting k3s with --node-ip=<ipv4>,<ipv6> would be the normal way to do that.

[brandond]

For agents you can get away with the IPs changing on reboots or the like, but servers expect them to be static for the life of the node in the cluster. Neither will properly handle the IPs changing out from under K3s while it's running.

[kelvie]

I'll have to test again, but I think agents can't have their IPs change either, as the iptables rules destined toward agents also use a stored IP address.

Anyway, I'll spin up another cluster with static IPv6 addresses and --node-ip and see what happens. Thanks for the info (and I hope this helps others in the future!)