CNI Portmap Plugin issue with Talos 1.7+, breaking pod-to-service communication

[donch]

Bug Report

Description

Since Talos 1.7.0, there's an inconsistency between the iptables mode used by kubelet (nft) and the one available for CNI plugins (legacy) when using the portmap plugin. This affects setups using CNIs that rely on the portmap plugin chaining.

A critical issue has been identified where pods using hostPort 443 prevent all cluster pods from accessing any Service IP on port 443, including the Kubernetes API server.

While we suspect this is related to the missing iptables-nft binary on the Talos host, we cannot confirm this is the root cause. However, the bug is consistently reproducible.

Environment

• Talos version: 1.7.0+ and 1.8.0
• Kubernetes version: v1.25+
• CNI: kube-router with portmap plugin
• Previous working version: Talos 1.6.x

Workaround

We have found a temporary workaround for clusters running Talos 1.7+ with kube-router:

• Remove all hostPort configurations from DaemonSets and other workloads
• For ingress controllers specifically:
  • Switch from hostPort to hostNetwork: true
• This workaround allows clusters to maintain functionality
• Note: This is not a solution but a temporary mitigation until the root cause is fixed

[smira]

I wonder if that's also the issue behind #9814 (or related).

[smira]

Do you have a reproducing configuration we could test with?

[donch]

Yes sure, the setup is quite simple :

• Deploy a Talos cluster with kube-proxy enabled in v1.7.0 with ipvs mode
• Install Kube-router from our chart we have tested with version 2.3.0 of the kube-router binary without success
• Deploy an nginx-ingress with hostPort enabled

You should see in the kube-router logs that it can't connect to the apiserver.

[smira]

If you have that as a series of shell commands and patches to Talos, that'd be perfect. E.g. I guess you disable Flannel.

A way for us to reproduce would make fixing this way easier.

[donch]

Yes sure, we use talhelper to generate Talos configuration :

clusterName: k8s
talosVersion: v1.8.3
kubernetesVersion: v1.28.15
endpoint: https://10.1.22.10:6443
domain: k8s.custom
clusterPodNets:
  - 10.1.128.0/20
clusterSvcNets:
  - 10.1.200.0/24
cniConfig:
  name: none
patches:
  - "@./nodes.patch.yaml"
controlPlane:
  patches:
    - "@./etcd.patch.yaml"
    - "@./cp.patch.yaml"
  schematic:
    customization:
      systemExtensions:
        officialExtensions:
          - siderolabs/qemu-guest-agent
          - siderolabs/zfs
worker:
  patches:
    - "@./workers.patch.yaml"
  schematic:
    customization:
      systemExtensions:
        officialExtensions:
          - siderolabs/qemu-guest-agent
          - siderolabs/zfs
nodes:
...

and the interesting patches

machine:
  features:
    kubePrism:
      port: 7443
  kernel:
    modules:
      - name: zfs
  sysctls:
    user.max_user_namespaces: "11255"
  systemDiskEncryption:
    ephemeral:
      provider: luks2
      keys:
        - nodeID: {}
          slot: 0
    state:
      provider: luks2
      keys:
        - nodeID: {}
          slot: 0
cluster:
  coreDNS:
    disabled: true
  apiServer:
    admissionControl:
      - name: PodSecurity
        configuration:
          exemptions:
            namespaces:
              - qemu-guest-agent
  proxy:
    mode: ipvs

The others are about system-reserved resources. We also disable CoreDNS since we use a custom configuration

For the kube-router install, the following would be enough :
helm upgrade --install --namespace kube-system kube-router kube-router --repo https://charts.enix.io/ --set kubeRouter.cni.install=true

At this point, all should be working since there is not yet any workloads with hostPort enabled.
You can install an ingress-nginx :
helm upgrade --install --create-namespace --namespace ingress-nginx ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --set controller.hostPort.enabled=true

Once deployed the bug happen, you can try to restart kube-router pods and see connection failure to the apiserver in the logs

[smira]

Thanks for detailed configuration, I confirmed that siderolabs/pkgs#1106 fixes ingress-nginx (it doesn't start without this fix properly).

[smira]

This doesn't fix #9814 though.