[QUESTION] Creating a cluster with tainted server nodes and untainted worker nodes has traefik install job fail

[dmusicant-dk]

What did you do

• How was the cluster created?

# Created the cluster with 3 master nodes that prevent non-control plane pods from installing
k3d cluster create dev --api-port 6551 --port "8081:80@loadbalancer" --servers 3 --k3s-server-arg --node-taint="CriticalAddonsOnly=true:NoExecute"`

# Created Worker nodes that do allow installing any pod
k3d node create worker --cluster dev --replicas 3

• What did you do afterwards?
  • k3d commands? No
  • docker commands? No
  • OS operations (e.g. shutdown/reboot)? No

What did you expect to happen

Everything would install properly that needs to install and k3d, knowing I was putting a taint on the server nodes would apply tolerations as needed to control plan pieces.

What actually happened?

When k3d tried to install traefik, it failed with:

1 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.

Screenshots or terminal output

# Ran this to check what pods are installed and you can see it's not running
kubectl get jobs -n kube-system
# NAME                   COMPLETIONS   DURATION   AGE
# helm-install-traefik   0/1           5m37s      5m48s

# You can see the events show the traefik pod cannot install
kubectl get events -n kube-system
# LAST SEEN   TYPE      REASON              OBJECT                                        MESSAGE
# 5m49s       Warning   FailedScheduling    pod/helm-install-traefik-ntxkr                0/1 nodes are available: 1 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.
# 5m47s       Warning   FailedScheduling    pod/helm-install-traefik-ntxkr                0/1 nodes are available: 1 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.
# 5m31s       Warning   FailedScheduling    pod/helm-install-traefik-ntxkr                0/2 nodes are available: 2 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.
# 5m9s        Warning   FailedScheduling    pod/helm-install-traefik-ntxkr                0/3 nodes are available: 3 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.
# 5m50s       Normal    SuccessfulCreate    job/helm-install-traefik                      Created pod: helm-install-traefik-ntxkr

Which OS & Architecture

• Ubuntu Linux 20.04

Which version of k3d

k3d version v3.4.0
k3s version v1.19.4-k3s1 (default)

Which version of docker

Client: Docker Engine - Community
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:59:40 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:57:45 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Did you try any fixes?

Yes, I tried the fix suggested here which is to do a docker system prune -a but it didn't work (even after deleting the cluster, pruning and recreating).

The problem is that k3d uses Helm to install traefik so I can't edit the yaml to add in a toleration. But I also don't understand why it never tries to install on the other nodes.

[iwilltry42]

Hi @dmusicant-dk , I just tried to reproduce this with your commands (just removing the trailing ` in the first one, as I assume that's a copy-paste leftover) and it works just fine:

$ k3d cluster create dev --api-port 6551 --port "8081:80@loadbalancer" --servers 3 --k3s-server-arg --node-taint="CriticalAddonsOnly=true:NoExecute"
...

$ k3d node create worker --cluster dev --replicas 3
...

$ watch kubectl get nodes # waiting for all nodes to be "Ready"
...

$ kubectl describe pod helm-install-traefik-4brss
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  67s   default-scheduler  0/1 nodes are available: 1 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.
  Warning  FailedScheduling  67s   default-scheduler  0/1 nodes are available: 1 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.
  Warning  FailedScheduling  51s   default-scheduler  0/2 nodes are available: 2 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.
  Warning  FailedScheduling  31s   default-scheduler  0/3 nodes are available: 3 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate.
  Normal   Scheduled         11s   default-scheduler  Successfully assigned kube-system/helm-install-traefik-4brss to k3d-worker-2
  Normal   Pulling           8s    kubelet            Pulling image "rancher/klipper-helm:v0.3.2"

$ kubectl get all --all-namespaces

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-854c77959c-p95kt                  1/1     Running     0          80s
kube-system   pod/helm-install-traefik-4brss                0/1     Completed   0          80s
kube-system   pod/local-path-provisioner-7c458769fb-nnmfb   1/1     Running     0          80s
kube-system   pod/metrics-server-86cbb8457f-bwbz2           1/1     Running     0          80s
kube-system   pod/svclb-traefik-2n9g9                       2/2     Running     0          12s
kube-system   pod/svclb-traefik-bxfxk                       2/2     Running     0          12s
kube-system   pod/svclb-traefik-d766j                       2/2     Running     0          12s
kube-system   pod/svclb-traefik-dpx8h                       2/2     Running     0          12s
kube-system   pod/svclb-traefik-f4kbs                       2/2     Running     0          12s
kube-system   pod/svclb-traefik-hmmx6                       2/2     Running     0          12s
kube-system   pod/traefik-6f9cbd9bd4-2kxw6                  0/1     Running     0          12s

NAMESPACE     NAME                         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
default       service/kubernetes           ClusterIP      10.43.0.1      <none>        443/TCP                      96s
kube-system   service/kube-dns             ClusterIP      10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP       93s
kube-system   service/metrics-server       ClusterIP      10.43.19.0     <none>        443/TCP                      93s
kube-system   service/traefik              LoadBalancer   10.43.91.241   172.25.0.3    80:30895/TCP,443:30301/TCP   12s
kube-system   service/traefik-prometheus   ClusterIP      10.43.90.122   <none>        9100/TCP                     12s

NAMESPACE     NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/svclb-traefik   6         6         6       6            6           <none>          12s

NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns                  1/1     1            1           93s
kube-system   deployment.apps/local-path-provisioner   1/1     1            1           93s
kube-system   deployment.apps/metrics-server           1/1     1            1           93s
kube-system   deployment.apps/traefik                  0/1     1            0           12s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-854c77959c                  1         1         1       81s
kube-system   replicaset.apps/local-path-provisioner-7c458769fb   1         1         1       81s
kube-system   replicaset.apps/metrics-server-86cbb8457f           1         1         1       81s
kube-system   replicaset.apps/traefik-6f9cbd9bd4                  1         1         0       13s

NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   1/1           69s        94s

Maybe it just needs some more time? 🤔
Are all nodes up and ready in your case?

BTW: This would rather be a case for the new GitHub Discussions feature (however, I couldn't migrate this issue there, as the button somehow went missing 🤷‍♂️)