RKE2 1.27.6 installation stuck in ContainerCreating: accessing container registry instead of kubernetes svc

[andreas-p]

Environmental Info:
RKE2 Version: v1.27.6+rke2r1

Node(s) CPU architecture, OS, and Version:

Linux 6.1.0-12-amd64 Debian12

Cluster Configuration:

first node, fresh air-gapped install from tarballs on empty machine (/var/lib/cni and /var/lib/rancher cleaned, iptables installed), minimal configuration:

node-name: this-node
tls-san:
- haproxy
- this-node
token-file: /etc/rancher/rke2/token

Environment:

CONTAINERD_HTTPS_PROXY=http://registry-proxy:3128
CONTAINERD_HTTP_PROXY=http://registry-proxy:3128
CONTAINERD_NO_PROXY=127.0.0.1,.my-domain

Describe the bug:

After the service is started and kubectl reports the node as ready, some pods are still not up:

cloud-controller-manager-kht10                          1/1     Running             0          114s
etcd-kht10                                              1/1     Running             0          105s
helm-install-rke2-canal-jsdvm                           0/1     Completed           0          2m2s
helm-install-rke2-coredns-kcpcm                         0/1     Completed           0          2m2s
helm-install-rke2-ingress-nginx-7szq7                   0/1     ContainerCreating   0          2m2s
helm-install-rke2-metrics-server-mtv4b                  0/1     ContainerCreating   0          2m2s
helm-install-rke2-snapshot-controller-crd-j4zw5         0/1     ContainerCreating   0          2m1s
helm-install-rke2-snapshot-controller-tbznh             0/1     ContainerCreating   0          2m2s
helm-install-rke2-snapshot-validation-webhook-lbnz6     0/1     ContainerCreating   0          2m1s
kube-apiserver-kht10                                    1/1     Running             0          84s
kube-controller-manager-kht10                           1/1     Running             0          116s
kube-proxy-kht10                                        1/1     Running             0          118s
kube-scheduler-kht10                                    1/1     Running             0          116s
rke2-canal-6g5jd                                        2/2     Running             0          90s
rke2-coredns-rke2-coredns-5f5d6b54c7-9m4nv              0/1     ContainerCreating   0          95s
rke2-coredns-rke2-coredns-autoscaler-6bf8f59fd5-k4xgc   0/1     ContainerCreating   0          95s

kubectl describe pod rke2-coredns-rke2-coredns-5f5d6b54c7-9m4nv will show the problem:

Warning  FailedCreatePodSandBox  2m16s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f3d651b61620af99367edceeb836f3683c80e59eb9301fee1c9849d2f8cfa199": plugin type="calico" failed (add): error getting ClusterInformation: Get "https://10.43.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": tls: failed to verify certificate: x509: certificate is valid for 192.168.17.101, 127.0.0.1, ::1, not 10.43.0.1
Normal   SandboxChanged          5s (x11 over 2m15s)  kubelet            Pod sandbox changed, it will be killed and re-created.

The IP address mentioned in the error message, 192.168.17.101, is not configured anywhere on the machine; instead, it's the IP of the proxy to the private registry that is configured via CONTAINERD_HTTPS_PROXY=http://registry-proxy:3128.
Apparently, instead of accessing the kubernetes service, the container registry is contacted.

[andreas-p]

Found what's going wrong: https://docs.rke2.io/advanced?_highlight=containerd_#configuring-an-http-proxy is not correct about NO_PROXY settings and CONTAINERD_xxx. Will create a new issue for this tomorrow.