Enabling the new traefik support for existing cluster doesn't seem to be working - issues with traefik helm charts

[GameBurrow]

Environmental Info:
RKE2 Version:

zsh ❯ rke2 -v
rke2 version v1.31.0+rke2r1 (4bf1156f41631dc0e81d66a6834a53d14b48aaef)
go version go1.22.5 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
All servers running Debian 12, use amd64 type CPUs.

Cluster Configuration:

1 master node, 2 worker nodes. Master node is scheduled not to have pods.

NAME                    STATUS   ROLES                       AGE    VERSION          INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
sepaurg-kube-master-1   Ready    control-plane,etcd,master   138d   v1.31.0+rke2r1   [omitted]   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-25-amd64   containerd://1.7.20-k3s1
sepaurg-kube-worker-1   Ready    control-plane,etcd,master   136d   v1.31.0+rke2r1   [omitted]    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-20-amd64   containerd://1.7.20-k3s1
sepaurg-servers2        Ready    control-plane,etcd,master   138d   v1.31.0+rke2r1   [omitted]     <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-20-amd64   containerd://1.7.20-k3s1

Describe the bug:

Tried adding the new option ingress-controller: traefik to the rke2 config file:

cat /etc/rancher/rke2/config.yaml
ingress-controller: traefik

tls-san:
  - [omitted]

Restarted rke2-server services on all servers as my servers are already running. Traefik ingress doesn't seem to pop up. Looking at the logs I see following:

zsh ❯ journalctl -u rke2-server | grep traefik
Sep 16 15:07:23 sepaurg-kube-master-1 rke2[2769708]: time="2024-09-16T15:07:23Z" level=info msg="Extracting file charts/rke2-traefik-crd.yaml to /var/lib/rancher/rke2/data/v1.31.0-rke2r1-132800c9f107/charts/rke2-traefik-crd.yaml"
Sep 16 15:07:23 sepaurg-kube-master-1 rke2[2769708]: time="2024-09-16T15:07:23Z" level=info msg="Extracting file charts/rke2-traefik.yaml to /var/lib/rancher/rke2/data/v1.31.0-rke2r1-132800c9f107/charts/rke2-traefik.yaml"
Sep 16 15:07:23 sepaurg-kube-master-1 rke2[2769708]: time="2024-09-16T15:07:23Z" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-traefik-crd.yaml to set cluster configuration values"
Sep 16 15:07:23 sepaurg-kube-master-1 rke2[2769708]: time="2024-09-16T15:07:23Z" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-traefik.yaml to set cluster configuration values"
Sep 16 15:10:23 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:10:23Z" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-traefik-crd.yaml to set cluster configuration values"
Sep 16 15:10:23 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:10:23Z" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-traefik.yaml to set cluster configuration values"
Sep 16 15:20:00 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:00Z" level=error msg="error syncing 'kube-system/rke2-traefik-crd': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik-crd by helm-delete-rke2-traefik-crd, requeuing"
Sep 16 15:20:00 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:00Z" level=error msg="error syncing 'kube-system/rke2-traefik': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik by helm-delete-rke2-traefik, requeuing"
Sep 16 15:20:03 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:03Z" level=error msg="error syncing 'kube-system/rke2-traefik-crd': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik-crd by helm-delete-rke2-traefik-crd, requeuing"
Sep 16 15:20:03 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:03Z" level=error msg="error syncing 'kube-system/rke2-traefik': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik by helm-delete-rke2-traefik, requeuing"
Sep 16 15:20:06 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:06Z" level=error msg="error syncing 'kube-system/rke2-traefik-crd': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik-crd by helm-delete-rke2-traefik-crd, requeuing"
Sep 16 15:20:06 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:06Z" level=error msg="error syncing 'kube-system/rke2-traefik': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik by helm-delete-rke2-traefik, requeuing"
Sep 16 15:20:09 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:09Z" level=error msg="error syncing 'kube-system/rke2-traefik-crd': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik-crd by helm-delete-rke2-traefik-crd, requeuing"
Sep 16 15:20:09 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:09Z" level=error msg="error syncing 'kube-system/rke2-traefik': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik by helm-delete-rke2-traefik, requeuing"
Sep 16 15:20:12 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:12Z" level=error msg="error syncing 'kube-system/rke2-traefik-crd': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik-crd by helm-delete-rke2-traefik-crd, requeuing"
Sep 16 15:20:15 sepaurg-kube-master-1 rke2[852]: time="2024-09-16T15:20:15Z" level=error msg="error syncing 'kube-system/rke2-traefik-crd': handler on-helm-chart-remove: waiting for delete of helm chart for kube-system/rke2-traefik-crd by helm-delete-rke2-traefik-crd, requeuing"

Steps To Reproduce:

1. Cluster has been running for a while, using the built-in nginx ingress
2. Added the new option to master server config file /etc/rancher/rke2/config.yaml
3. Restarted rke2-server services on all servers

[brandond]

Tried adding the new option ingress-controller: traefik to the rke2 config file:
Cluster has been running for a while, using the built-in nginx ingress

We do not currently supporting changing the ingress controller on a running cluster, in the same way that we do not support changing the CNI on a running cluster.