rke2-canal: CALICO_NETWORKING_BACKEND none creates noisy log output

[curantes]

Environmental Info:
RKE2 Version:
rke2 version v1.27.5+rke2r1 (1de9953)
go version go1.20.7 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
Linux k8s-dev001-master001.foo.bar.com 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

Describe the bug:

log gets noisy from rke2-canal (calico) in version 3.26.1 and there is an upstream issue for this and they have also released a fix for this in v3.26.3. Please upgrade to v3.26.3 and backport this fix to previous still supported releases of rke2.

linked issue: projectcalico/calico#7819

As of now we get 23 million extra logs for all our clusters each day.

Steps To Reproduce:

• Installed RKE2:

Expected behavior:

Actual behavior:

Additional context / logs:

The log thats get spammed: Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.

[thomasferrandiz]

QA Validation steps:

• Create /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml with the following content:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    calico:
      networkingBackend: none

• install and start rke2-server.

• check that the line does not appear in the log every 1s:

Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.

[StefVO]

Can this be back ported to v1.27.X as well?

[endawkins]

Tested on v1.28.4, v1.28.5, v1.28.6

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-26-180 6.2.0-1017-aws #17~22.04.1-Ubuntu SMP Fri Nov 17 21:07:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Cluster Configuration:

1 server
1 agent

Config.yaml:

server config.yaml:

write-kubeconfig-mode: 644
token: test
node-external-ip: <EXTERNAL_IP>

agent config.yaml:

token: test
server: "https://<SERVER_1_IP_ADDRESS>:9345"

Additional files

rke2-canal-config.yaml:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    calico:
      networkingBackend: none

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/rke2/ && sudo cp config.yaml /etc/rancher/rke2/ && cat /etc/rancher/rke2/config.yaml

2. Create the rke2-canal-config file with the rke2-canal-config.yaml contents above:

$ sudo mkdir -p /var/lib/rancher/rke2/server/manifests/
$ sudo vi /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml

3. Install RKE2

$ sudo systemctl enable rke2-server.service && sudo systemctl start rke2-server.service
$ sudo systemctl enable rke2-agent.service && sudo systemctl start rke2-agent.service

4. Verify that the following line does not display in the logs for the rke2-canal pod:

$ kubectl logs <resource_name> -n <namespace>
Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.

v1.28.4+rke2r1 Results:

• rke2 version used for v1.28.4+rke2r1:

$ rke2 -v
rke2 version v1.28.4+rke2r1 (f91b98a64660674406589c2b30fead7ca21d6b51)
go version go1.20.11 X:boringcrypto

$ kubectl logs pod/rke2-canal-gh95r -n kube-system | grep "Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release."
Defaulted container "calico-node" out of: calico-node, kube-flannel, install-cni (init), flexvol-driver (init)
W0125 21:13:39.547994       9 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:13:39.683725      17 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:13:41.047623      44 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:13:41.120898      45 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:13:41.137271      46 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:13:41.199255      47 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.

$ kubectl logs pod/rke2-canal-hh86l -n kube-system | grep "Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release."
Defaulted container "calico-node" out of: calico-node, kube-flannel, install-cni (init), flexvol-driver (init)
W0125 21:14:59.196424       9 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:14:59.356751      17 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:15:00.697236      46 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:15:00.699717      44 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:15:00.702362      45 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:15:00.740541      47 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.

Additional context/logs

$ kubectl get nodes,pods -A -o wide                                                                                                                                                        ip-172-31-22-59: Thu Jan 25 21:27:25 2024

NAME                    STATUS   ROLES                       AGE   VERSION          INTERNAL-IP     EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
node/ip-172-31-16-206   Ready    <none>                      13m   v1.28.4+rke2r1   172.31.16.206   <none>          Ubuntu 22.04.3 LTS   6.2.0-1017-aws   containerd://1.7.7-k3s1
node/ip-172-31-22-59    Ready    control-plane,etcd,master   14m   v1.28.4+rke2r1   172.31.22.59    <EXTERNAL_IP>   Ubuntu 22.04.3 LTS   6.2.0-1017-aws   containerd://1.7.7-k3s1

NAMESPACE     NAME                                                       READY   STATUS      RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/cloud-controller-manager-ip-172-31-22-59               1/1     Running     0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/etcd-ip-172-31-22-59                                   1/1     Running     0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/helm-install-rke2-canal-l6dhc                          0/1     Completed   0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/helm-install-rke2-coredns-csrwj                        0/1     Completed   0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/helm-install-rke2-ingress-nginx-49whq                  0/1     Completed   0          14m   10.42.0.4       ip-172-31-22-59    <none>           <none>
kube-system   pod/helm-install-rke2-metrics-server-j6dnw                 0/1     Completed   0          14m   10.42.0.5       ip-172-31-22-59    <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-crd-sp498        0/1     Completed   0          14m   10.42.0.7       ip-172-31-22-59    <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-j4gl8            0/1     Completed   1          14m   10.42.0.8       ip-172-31-22-59    <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-75wpg    0/1     Completed   0          14m   10.42.0.3       ip-172-31-22-59    <none>           <none>
kube-system   pod/kube-apiserver-ip-172-31-22-59                         1/1     Running     0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/kube-controller-manager-ip-172-31-22-59                1/1     Running     0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-16-206                            1/1     Running     0          13m   172.31.16.206   ip-172-31-16-206   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-22-59                             1/1     Running     0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/kube-scheduler-ip-172-31-22-59                         1/1     Running     0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/rke2-canal-gh95r                                       2/2     Running     0          14m   172.31.22.59    ip-172-31-22-59    <none>           <none>
kube-system   pod/rke2-canal-hh86l                                       2/2     Running     0          13m   172.31.16.206   ip-172-31-16-206   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-6b795db654-5fddc             1/1     Running     0          14m   10.42.0.2       ip-172-31-22-59    <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-6b795db654-85hdm             1/1     Running     0          13m   10.42.1.2       ip-172-31-16-206   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-945fbd459-sgz82   1/1     Running     0          14m   10.42.0.6       ip-172-31-22-59    <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-6d6x4                    1/1     Running     0          12m   10.42.1.3       ip-172-31-16-206   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-lvp5f                    1/1     Running     0          13m   10.42.0.14      ip-172-31-22-59    <none>           <none>
kube-system   pod/rke2-metrics-server-544c8c66fc-pxrqn                   1/1     Running     0          13m   10.42.0.9       ip-172-31-22-59    <none>           <none>
kube-system   pod/rke2-snapshot-controller-59cc9cd8f4-hgb46              1/1     Running     0          13m   10.42.0.12      ip-172-31-22-59    <none>           <none>
kube-system   pod/rke2-snapshot-validation-webhook-54c5989b65-q8bjf      1/1     Running     0          13m   10.42.0.10      ip-172-31-22-59    <none>           <none>

v1.28.5+rke2r1 Results:

• rke2 version used for v1.28.5+rke2r1:

$ rke2 -v
rke2 version v1.28.5+rke2r1 (adcd936ed6d7280080fba3670208cdea532de453)
go version go1.20.12 X:boringcrypto

$ kubectl logs pod/rke2-canal-94vpp -n kube-system | grep "Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release."
Defaulted container "calico-node" out of: calico-node, kube-flannel, install-cni (init), flexvol-driver (init)
W0125 02:00:19.856394      10 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 02:00:19.939491      18 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 02:00:21.205060      46 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 02:00:21.205858      45 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 02:00:21.283532      47 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 02:00:21.286348      48 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.

v1.28.6+rke2r1 Results:

• rke2 version used for v1.28.6+rke2r1:

$ rke2 -v
rke2 version v1.28.6-rc2+rke2r1 (26d0a97e92fcf5e2786784b30d96096bbeb9d537)
go version go1.20.13 X:boringcrypto

$ kubectl logs pod/rke2-canal-mr722 -n kube-system | grep "Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release."
Defaulted container "calico-node" out of: calico-node, kube-flannel, install-cni (init), flexvol-driver (init)
W0125 18:29:30.716815       8 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 18:29:30.869826      16 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 18:29:32.584886      46 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 18:29:32.651926      47 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 18:29:32.687682      44 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 18:29:32.798875      45 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.

$ kubectl logs pod/rke2-canal-knbvv -n kube-system | grep "Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release."
Defaulted container "calico-node" out of: calico-node, kube-flannel, install-cni (init), flexvol-driver (init)
W0125 21:14:49.928149      10 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:14:50.061817      18 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:14:51.420409      47 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:14:51.423846      46 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:14:51.454669      48 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
W0125 21:14:51.490100      45 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.

Additional context / logs:

$ kubectl get nodes,pods -A -o wide                                                                                                                                                       ip-172-31-23-251: Thu Jan 25 21:25:35 2024

NAME                    STATUS   ROLES                       AGE    VERSION          INTERNAL-IP     EXTERNAL-IP    OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
node/ip-172-31-23-251   Ready    control-plane,etcd,master   177m   v1.28.6+rke2r1   172.31.23.251   <EXTERNAL_IP>  Ubuntu 22.04.3 LTS   6.2.0-1017-aws   containerd://1.7.11-k3s2
node/ip-172-31-24-15    Ready    <none>                      11m    v1.28.6+rke2r1   172.31.24.15    <none>         Ubuntu 22.04.3 LTS   6.2.0-1017-aws   containerd://1.7.11-k3s2

NAMESPACE     NAME                                                        READY   STATUS      RESTARTS   AGE    IP              NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/cloud-controller-manager-ip-172-31-23-251               1/1     Running     0          177m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/etcd-ip-172-31-23-251                                   1/1     Running     0          176m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/helm-install-rke2-canal-hjd77                           0/1     Completed   0          176m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/helm-install-rke2-coredns-vkzhf                         0/1     Completed   0          176m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/helm-install-rke2-ingress-nginx-4r956                   0/1     Completed   0          176m   10.42.0.7       ip-172-31-23-251   <none>           <none>
kube-system   pod/helm-install-rke2-metrics-server-njfdw                  0/1     Completed   0          176m   10.42.0.4       ip-172-31-23-251   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-crd-rqcd7         0/1     Completed   0          176m   10.42.0.6       ip-172-31-23-251   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-cxz6k             0/1     Completed   0          176m   10.42.0.8       ip-172-31-23-251   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-c7lr8     0/1     Completed   0          176m   10.42.0.2       ip-172-31-23-251   <none>           <none>
kube-system   pod/kube-apiserver-ip-172-31-23-251                         1/1     Running     0          177m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/kube-controller-manager-ip-172-31-23-251                1/1     Running     0          177m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-23-251                             1/1     Running     0          177m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-24-15                              1/1     Running     0          11m    172.31.24.15    ip-172-31-24-15    <none>           <none>
kube-system   pod/kube-scheduler-ip-172-31-23-251                         1/1     Running     0          177m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/rke2-canal-knbvv                                        2/2     Running     0          11m    172.31.24.15    ip-172-31-24-15    <none>           <none>
kube-system   pod/rke2-canal-mr722                                        2/2     Running     0          176m   172.31.23.251   ip-172-31-23-251   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-78f5dbb485-nv9dc              1/1     Running     0          11m    10.42.1.2       ip-172-31-24-15    <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-78f5dbb485-qvfwh              1/1     Running     0          176m   10.42.0.5       ip-172-31-23-251   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-64b867c686-lwx6m   1/1     Running     0          176m   10.42.0.3       ip-172-31-23-251   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-dr7sm                     1/1     Running     0          175m   10.42.0.13      ip-172-31-23-251   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-hh8gz                     1/1     Running     0          10m    10.42.1.3       ip-172-31-24-15    <none>           <none>
kube-system   pod/rke2-metrics-server-544c8c66fc-6swm4                    1/1     Running     0          175m   10.42.0.11      ip-172-31-23-251   <none>           <none>
kube-system   pod/rke2-snapshot-controller-59cc9cd8f4-6hmwk               1/1     Running     0          175m   10.42.0.10      ip-172-31-23-251   <none>           <none>
kube-system   pod/rke2-snapshot-validation-webhook-54c5989b65-pdjvk       1/1     Running     0          175m   10.42.0.9       ip-172-31-23-251   <none>           <none>

Conclusion
The issue has not been fixed per observations in the logs across 1.28.4, 1.28.5, and 1.28.6

[thomasferrandiz]

After checking again the previous version (v1.28.3 with calico v3.26.1), it appears that the fix is working as intended.

With v1.28.3, the line is logged every 1s forever which creates thousands of lines of logs per day whereas in v1.28.4 and later the line appears only 6 times when the pod starts and is not logged after.

Sorry about that, I should have been more explicit in the description of the validation steps.

[endawkins]

Per conversation with Thomas - the previous comment with observations for 1.28.4, 1.28.5, and 1.28.6 serve as validation.