restrict pod<->pod traffic to a certain NIC on the host (Using Calico)

[mhegreberg]

Environmental Info:
RKE2 Version:

rke2 version v1.26.15+rke2r1 (a413a7f)
go version go1.21.8 X:boringcrypto

Node(s) CPU architecture, OS, and Version:

Linux rke2-prod1-worker1 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

Cluster Configuration:

3 server nodes, 2 worker nodes. all nodes live on the same subnet, but the worker nodes have a second NIC in a different subnet for communicating to our SAN for pvcs

Describe the bug:

pod-to pod traffic seems to be communicating across both NICs on our worker nodes. I'd really like to restrict pod-pod traffic to only one NIC. I've messed with the routing tables on the nodes to block force the nodes to not see each other on the SAN subnet, but then I start getting CoreDNS failures and other pod-pod issues. Is there an easy way to tell calico that all pod-pod traffic needs to go out a certain NIC/subnet? they have the same name on all nodes if that's helpful

[manuelbuil]

If you set this config https://docs.tigera.io/calico/latest/reference/installation/api#operator.tigera.io/v1.NodeAddressAutodetection it will pick the interface you set

[mhegreberg]

So I can set it to allow pod-> pod traffic on one NIC, and allow external traffic to just use another/the routing table on the host node?

[manuelbuil]

pod traffic will use the nic you pick because the routing table will be set accordingly. The rest of the traffic will pick one interface based on the routing tables

[mhegreberg]

that sounds like exactly what I'm looking for!

within rke2/rancher, what is the recommended way of modifying system charts like this? do I do a helm upgrade and modify the values file for rke2-calico or will rke2 wipe that out next time there's a k8s upgrade?

[mhegreberg]

I was able to set this in the cluster config in rancher, and the setting populated. I believe it's working but I'll continue to test.

thanks!

[manuelbuil]

great!