Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After restart of pods, one of the Canal pod is getting a Init:CreateContainerConfigError #4742

Closed
est-suse opened this issue Sep 8, 2023 · 11 comments
Closed

After restart of pods, one of the Canal pod is getting a Init:CreateContainerConfigError #4742

est-suse opened this issue Sep 8, 2023 · 11 comments
Assignees
@manuelbuil @rbrtbnfgl

Comments

@est-suse
Copy link
Contributor

est-suse commented Sep 8, 2023 

v1.28.1-rc2+rke2r1

Node(s) CPU architecture, OS, and Version:

NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp4"
DOCUMENTATION_URL="https://documentation.suse.com/"

Cluster Configuration:

3 servers - 1 agent

Describe the bug:

After the performing of a “Cluster reset” test, one of the Canal Pod is getting a Init:CreateContainerConfigError

Steps To Reproduce:

1. Create an HA cluster, make sure it's up and ready
2. Deploy some workloads.
3. Stop two server nodes by using "sudo rke2-killall.sh"
4. Make sure the cluster is not accessible now
5. Shut down the rke2 server on that remaining node by running "sudo systemctl stop rke2-server"
6. Now run "sudo rke2 server --cluster-reset"
7. After the command completes, restart the rke2 server process: "sudo systemctl start rke2-server"
8. Check to get it working again and after a while the other nodes will not be ready. Usually have to wait a few minutes for this to occur.
9. Remove the db directories from other servers by running "sudo rm -rf /var/lib/rancher/rke2/server/db"
10. Restart the rke2-server process on the other servers by running "sudo systemctl start rke2-server"

After following the steps, the error is displayed:

kube-system   rke2-canal-785mp                                                     2/2     Running                           2 (4m5s ago)    48m
kube-system   rke2-canal-h8lq6                                                     2/2     Running                           0               52m
kube-system   rke2-canal-nqb5t                                                     2/2     Running                           0               45m
kube-system   rke2-canal-qz4zh                                                     1/2     Init:CreateContainerConfigError   2 (80s ago)     49m
@est-suse
Copy link
Contributor Author

est-suse commented Sep 8, 2023

s1_kubelet.log
s2_kubelet.log
s3_kubelet.log
pods_server3.zip
pods_server2.zip
s1.txt
s2.txt
s3.txt
pods_server1.zip

@est-suse
Copy link
Contributor Author

est-suse commented Sep 8, 2023 •
edited by brandond
Loading 

 kubectl describe pod -n kube-system   rke2-canal-qz4zh
Name:                 rke2-canal-qz4zh
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      canal
Node:                 ip-172-31-3-82.us-east-2.compute.internal/172.31.3.82
Start Time:           Fri, 08 Sep 2023 17:46:00 +0000
Labels:               controller-revision-hash=8468c96957
                      k8s-app=canal
                      pod-template-generation=1
Annotations:          <none>
Status:               Running
IP:                   172.31.3.82
IPs:
  IP:           172.31.3.82
Controlled By:  DaemonSet/rke2-canal
Init Containers:
  install-cni:
    Container ID:
    Image:         rancher/hardened-calico:v3.26.1-build20230802
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Waiting
      Reason:       CreateContainerConfigError
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 08 Sep 2023 17:47:01 +0000
      Finished:     Fri, 08 Sep 2023 17:47:07 +0000
    Ready:          False
    Restart Count:  1
    Environment:
      CALICO_CNI_SERVICE_ACCOUNT:   (v1:spec.serviceAccountName)
      CNI_CONF_NAME:               10-canal.conflist
      CNI_NETWORK_CONFIG:          <set to the key 'cni_network_config' of config map 'rke2-canal-config'>  Optional: false
      KUBERNETES_NODE_NAME:         (v1:spec.nodeName)
      CNI_MTU:                     <set to the key 'veth_mtu' of config map 'rke2-canal-config'>  Optional: false
      SLEEP:                       false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nlbgf (ro)
  flexvol-driver:
    Container ID:  containerd://1ca4c9763e2885dab5ae9c69572135f405eddbdf700953b53b766dabe3e8b3dc
    Image:         rancher/hardened-calico:v3.26.1-build20230802
    Image ID:      docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/flexvol.sh
      -s
      /usr/local/bin/flexvol
      -i
      flexvoldriver
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 08 Sep 2023 17:47:08 +0000
      Finished:     Fri, 08 Sep 2023 17:47:08 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nlbgf (ro)
Containers:
  calico-node:
    Container ID:  containerd://96846bb6801317be90e0160c55beed417026627e1f178adbd9f8000125a19555
    Image:         rancher/hardened-calico:v3.26.1-build20230802
    Image ID:      docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c
    Port:          <none>
    Host Port:     <none>
    Command:
      start_runit
    State:          Running
      Started:      Fri, 08 Sep 2023 18:34:38 +0000
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Fri, 08 Sep 2023 17:47:09 +0000
      Finished:     Fri, 08 Sep 2023 18:33:45 +0000
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:      250m
    Liveness:   exec [/usr/bin/calico-node -felix-live] delay=10s timeout=10s period=10s #success=1 #failure=6
    Readiness:  http-get http://localhost:9099/readiness delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:                     kubernetes
      USE_POD_CIDR:                       true
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_CNI_SERVICE_ACCOUNT:          (v1:spec.serviceAccountName)
      CALICO_NETWORKING_BACKEND:          none
      CLUSTER_TYPE:                       k8s,canal
      FELIX_IPTABLESREFRESHINTERVAL:      60
      FELIX_IPTABLESBACKEND:              auto
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGSEVERITYSCREEN:            info
      FELIX_HEALTHENABLED:                true
      FELIX_PROMETHEUSMETRICSENABLED:     true
      FELIX_XDPENABLED:                   false
      FELIX_FAILSAFEINBOUNDHOSTPORTS:
      FELIX_FAILSAFEOUTBOUNDHOSTPORTS:
      IP_AUTODETECTION_METHOD:            first-found
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nlbgf (ro)
  kube-flannel:
    Container ID:  containerd://e8acd35d49d97313727c6e46884f521a96b19a9d030253d944427f77552b9805
    Image:         rancher/hardened-flannel:v0.22.1-build20230802
    Image ID:      docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/bin/flanneld
      --ip-masq
      --kube-subnet-mgr
    State:          Running
      Started:      Fri, 08 Sep 2023 18:35:40 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 08 Sep 2023 18:34:38 +0000
      Finished:     Fri, 08 Sep 2023 18:35:08 +0000
    Ready:          True
    Restart Count:  2
    Environment:
      POD_NAME:              rke2-canal-qz4zh (v1:metadata.name)
      POD_NAMESPACE:         kube-system (v1:metadata.namespace)
      FLANNELD_IFACE:        <set to the key 'canal_iface' of config map 'rke2-canal-config'>        Optional: false
      FLANNELD_IFACE_REGEX:  <set to the key 'canal_iface_regex' of config map 'rke2-canal-config'>  Optional: false
      FLANNELD_IP_MASQ:      <set to the key 'masquerade' of config map 'rke2-canal-config'>         Optional: false
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nlbgf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rke2-canal-config
    Optional:  false
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/volumeplugins/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  kube-api-access-nlbgf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason          Age                From               Message
  ----     ------          ----               ----               -------
  Normal   Scheduled       62m                default-scheduler  Successfully assigned kube-system/rke2-canal-qz4zh to ip-172-31-3-82.us-east-2.compute.internal
  Normal   Pulling         62m                kubelet            Pulling image "rancher/hardened-calico:v3.26.1-build20230802"
  Normal   Pulled          62m                kubelet            Successfully pulled image "rancher/hardened-calico:v3.26.1-build20230802" in 16.275s (16.275s including waiting)
  Normal   Created         61m (x2 over 62m)  kubelet            Created container install-cni
  Normal   Started         61m (x2 over 62m)  kubelet            Started container install-cni
  Normal   Pulled          61m                kubelet            Container image "rancher/hardened-calico:v3.26.1-build20230802" already present on machine
  Normal   Pulled          61m                kubelet            Container image "rancher/hardened-calico:v3.26.1-build20230802" already present on machine
  Normal   Created         61m                kubelet            Created container flexvol-driver
  Normal   Started         61m                kubelet            Started container flexvol-driver
  Normal   Pulled          61m                kubelet            Container image "rancher/hardened-calico:v3.26.1-build20230802" already present on machine
  Normal   Created         61m                kubelet            Created container calico-node
  Normal   Started         61m                kubelet            Started container calico-node
  Normal   Pulling         61m                kubelet            Pulling image "rancher/hardened-flannel:v0.22.1-build20230802"
  Normal   Pulled          61m                kubelet            Successfully pulled image "rancher/hardened-flannel:v0.22.1-build20230802" in 4.321s (4.321s including waiting)
  Normal   Created         61m                kubelet            Created container kube-flannel
  Normal   Started         61m                kubelet            Started container kube-flannel
  Warning  NodeNotReady    18m                node-controller    Node is not ready
  Normal   SandboxChanged  13m                kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          13m                kubelet            Error: services have not yet been read at least once, cannot construct envvars
  Normal   Pulled          13m                kubelet            Container image "rancher/hardened-calico:v3.26.1-build20230802" already present on machine
  Warning  Failed          13m                kubelet            Error: services have not yet been read at least once, cannot construct envvars
  Warning  Failed          13m                kubelet            Error: services have not yet been read at least once, cannot construct envvars
  Normal   Pulled          13m (x2 over 13m)  kubelet            Container image "rancher/hardened-calico:v3.26.1-build20230802" already present on machine
  Normal   Created         13m                kubelet            Created container calico-node
  Normal   Started         13m                kubelet            Started container calico-node
  Normal   Created         13m                kubelet            Created container kube-flannel
  Normal   Started         13m                kubelet            Started container kube-flannel
  Warning  Unhealthy       13m                kubelet            Liveness probe failed: calico/node is not ready: Felix is not live: Get "http://localhost:9099/liveness": dial tcp [::1]:9099: connect: connection refused
  Warning  Unhealthy  13m  kubelet  Liveness probe failed: calico/node is not ready: Felix is not live: Get "http://localhost:9099/liveness": dial tcp [::1]:9099: connect: connection refused
  Warning  Unhealthy  13m (x6 over 13m)  kubelet  Readiness probe failed: Get "http://localhost:9099/readiness": dial tcp [::1]:9099: connect: connection refused
  Warning  BackOff    12m (x3 over 13m)  kubelet  Back-off restarting failed container kube-flannel in pod rke2-canal-qz4zh_kube-system(46518c5e-0964-456d-bdf4-199ab638d14b)
  Normal   Pulled     12m (x3 over 13m)  kubelet  Container image "rancher/hardened-flannel:v0.22.1-build20230802" already present on machine

@ShylajaDevadiga
Copy link
Contributor

Seeing this Init:CreateContainerConfigError on reboot of the nodes without cluster-reset. Pods don't come back clean.

> kubectl get pods -A |grep canal
kube-system   helm-install-rke2-canal-jtb78                                          0/1     Completed                         0             47m
kube-system   rke2-canal-457s2                                                       2/2     Init:CreateContainerConfigError   2 (30m ago)   47m
kube-system   rke2-canal-6868c                                                       2/2     Running                           2 (30m ago)   43m
kube-system   rke2-canal-slst8                                                       2/2     Init:CreateContainerConfigError   2 (30m ago)   44m
kube-system   rke2-canal-tl6b4                                                       2/2     Running                           3 (28m ago)   43m
>

@brandond
Copy link
Member

brandond commented Sep 8, 2023

The hardened-flannel:v0.22.1-build20230802 install-cni InitContainer is failing for some reason, but the kubelet logs and pod description don't say why. Have asked for containerd.log; waiting for that to be available.

@brandond
Copy link
Member

brandond commented Sep 8, 2023 •
edited
Loading

Appears to be an issue with the CNI config; containerd log says:

time="2023-09-08T18:20:36.801044353Z" level=info msg="CreateContainer within sandbox \"910770ea71c2eda2b9816ac77307b48e1fb1dcf3282fe2db268d6dbc123fc122\" for &ContainerMetadata{Name:install-cni,Attempt:0,} returns container id \"b9da4205ba4fdb732fffb20828a6c4d928be5addb13003b5d5485eec775c4d96\""
time="2023-09-08T18:20:36.805287133Z" level=info msg="StartContainer for \"b9da4205ba4fdb732fffb20828a6c4d928be5addb13003b5d5485eec775c4d96\""
time="2023-09-08T18:20:36.974544420Z" level=info msg="StartContainer for \"b9da4205ba4fdb732fffb20828a6c4d928be5addb13003b5d5485eec775c4d96\" returns successfully"
time="2023-09-08T18:20:47.561396750Z" level=error msg="failed to reload cni configuration after receiving fs change event(WRITE         \"/etc/cni/net.d/calico-kubeconfig\")" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2023-09-08T18:20:48.025446572Z" level=info msg="shim disconnected" id=b9da4205ba4fdb732fffb20828a6c4d928be5addb13003b5d5485eec775c4d96 namespace=k8s.io
time="2023-09-08T18:20:48.025532414Z" level=warning msg="cleaning up after shim disconnected" id=b9da4205ba4fdb732fffb20828a6c4d928be5addb13003b5d5485eec775c4d96 namespace=k8s.io

level=error msg="failed to reload cni configuration after receiving fs change event(WRITE \"/etc/cni/net.d/calico-kubeconfig\")" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"

It seems to be upset that calico-kubeconfig is being dropped in /etc/cni/net.d, as it's not a valid CNI config file. I see that it exists there on 1.27 and earlier though, where we are running the same version of containerd, flannel, and so on - so I'm not sure why it's upset about it here.

@est-suse
Copy link
Contributor Author

est-suse commented Sep 8, 2023

s2_containerd.log
s1_containerd.log

@brandond
Copy link
Member

brandond commented Sep 8, 2023 •
edited
Loading

Would like @manuelbuil and @rbrtbnfgl to take a look at this as it seems related to canal's calico somehow.

I suspect that what's happening is that:

  1. The canal initContainer runs to drop the CNI configuration and kubeconfig
  2. Writing the updated kubeconfig triggers containerd to reload the CNI config (reload is triggerd by any change to files in that directory)
  3. Canal fails to reload because the apiserver isn't up yet

I am curious if dropping the kubeconfig somewhere other than that directory might fix that. Unfortunately it looks like it's hardcoded to that path, so changing that may be a dead-end; we probably need to figure out where the cni plugin not initialized: failed to load cni config error is coming from.

@brandond brandond changed the title After System cluster reset, one of the Canal pod is getting a Init:CreateContainerConfigError After restart of pods, one of the Canal pod is getting a Init:CreateContainerConfigError Sep 8, 2023
@brandond
Copy link
Member

brandond commented Sep 8, 2023 •
edited
Loading

1.27 shows the same error in the containerd log so maybe it's a red herring?

time="2023-09-08T19:29:32.923173967Z" level=info msg="CreateContainer within sandbox \"45a0505b89dc3979d4ba1a5bcc4d9c7c77be1eac4b6a426ce5649d556e7ec657\" for &ContainerMetadata{Name:install-cni,Attempt:0,} returns container id \"40d46a179a0978f35272bdbb8381c50cc021ce579debe8c9e553befb4aba4f80\""
time="2023-09-08T19:29:32.923856846Z" level=info msg="StartContainer for \"40d46a179a0978f35272bdbb8381c50cc021ce579debe8c9e553befb4aba4f80\""
time="2023-09-08T19:29:33.091354744Z" level=info msg="StartContainer for \"40d46a179a0978f35272bdbb8381c50cc021ce579debe8c9e553befb4aba4f80\" returns successfully"
time="2023-09-08T19:29:35.497388490Z" level=error msg="failed to reload cni configuration after receiving fs change event(WRITE         \"/etc/cni/net.d/calico-kubeconfig\")" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2023-09-08T19:29:36.431323635Z" level=info msg="shim disconnected" id=40d46a179a0978f35272bdbb8381c50cc021ce579debe8c9e553befb4aba4f80 namespace=k8s.io
time="2023-09-08T19:29:36.431402717Z" level=warning msg="cleaning up after shim disconnected"

@brandond
Copy link
Member

brandond commented Sep 8, 2023 •
edited
Loading

OK those are all red herrings; I'm not seeing anything new on the containerd side because the kubelet isn't even attempting to restart it. Setting kubelet-arg: v=6 gives the following:

E0908 21:11:23.582015    9799 kuberuntime_manager.go:1209] init container &Container{Name:install-cni,Image:rancher/hardened-calico:v3.26.1-build20230802,Command:[/opt/cni/bin/install],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:CALICO_CNI_SERVICE_ACCOUNT,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.serviceAccountName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:CNI_CONF_NAME,Value:10-canal.conflist,ValueFrom:nil,},EnvVar{Name:CNI_NETWORK_CONFIG,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:rke2-canal-config,},Key:cni_network_config,Optional:nil,},SecretKeyRef:nil,},},EnvVar{Name:KUBERNETES_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:CNI_MTU,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:rke2-canal-config,},Key:veth_mtu,Optional:nil,},SecretKeyRef:nil,},},EnvVar{Name:SLEEP,Value:false,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:cni-bin-dir,ReadOnly:false,MountPath:/host/opt/cni/bin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-net-dir,ReadOnly:false,MountPath:/host/etc/cni/net.d,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-qrxdq,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars
E0908 21:11:23.582137    9799 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"install-cni\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="kube-system/rke2-canal-4nblx" podUID="3a400dab-6eb5-47f3-b87d-2867f168bd7f"
E0908 21:11:24.172284    9799 kuberuntime_manager.go:1209] container &Container{Name:calico-node,Image:rancher/hardened-calico:v3.26.1-build20230802,Command:[start_runit],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:DATASTORE_TYPE,Value:kubernetes,ValueFrom:nil,},EnvVar{Name:USE_POD_CIDR,Value:true,ValueFrom:nil,},EnvVar{Name:WAIT_FOR_DATASTORE,Value:true,ValueFrom:nil,},EnvVar{Name:NODENAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:CALICO_CNI_SERVICE_ACCOUNT,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.serviceAccountName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:CALICO_NETWORKING_BACKEND,Value:none,ValueFrom:nil,},EnvVar{Name:CLUSTER_TYPE,Value:k8s,canal,ValueFrom:nil,},EnvVar{Name:FELIX_IPTABLESREFRESHINTERVAL,Value:60,ValueFrom:nil,},EnvVar{Name:FELIX_IPTABLESBACKEND,Value:auto,ValueFrom:nil,},EnvVar{Name:CALICO_DISABLE_FILE_LOGGING,Value:true,ValueFrom:nil,},EnvVar{Name:FELIX_DEFAULTENDPOINTTOHOSTACTION,Value:ACCEPT,ValueFrom:nil,},EnvVar{Name:FELIX_IPV6SUPPORT,Value:false,ValueFrom:nil,},EnvVar{Name:FELIX_LOGSEVERITYSCREEN,Value:info,ValueFrom:nil,},EnvVar{Name:FELIX_HEALTHENABLED,Value:true,ValueFrom:nil,},EnvVar{Name:FELIX_PROMETHEUSMETRICSENABLED,Value:true,ValueFrom:nil,},EnvVar{Name:FELIX_XDPENABLED,Value:false,ValueFrom:nil,},EnvVar{Name:FELIX_FAILSAFEINBOUNDHOSTPORTS,Value:,ValueFrom:nil,},EnvVar{Name:FELIX_FAILSAFEOUTBOUNDHOSTPORTS,Value:,ValueFrom:nil,},EnvVar{Name:IP_AUTODETECTION_METHOD,Value:first-found,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{250 -3} {<nil>} 250m DecimalSI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:cni-net-dir,ReadOnly:false,MountPath:/host/etc/cni/net.d,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:lib-modules,ReadOnly:true,MountPath:/lib/modules,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:var-run-calico,ReadOnly:false,MountPath:/var/run/calico,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:var-lib-calico,ReadOnly:false,MountPath:/var/lib/calico,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:policysync,ReadOnly:false,MountPath:/var/run/nodeagent,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-log-dir,ReadOnly:true,MountPath:/var/log/calico/cni,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-qrxdq,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[/usr/bin/calico-node -felix-live],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:10,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:6,TerminationGracePeriodSeconds:nil,},ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readiness,Port:{0 9099 },Host:localhost,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/usr/bin/calico-node -shutdown],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars
E0908 21:11:24.173166    9799 kuberuntime_manager.go:1209] container &Container{Name:kube-flannel,Image:rancher/hardened-flannel:v0.22.1-build20230802,Command:[/opt/bin/flanneld --ip-masq --kube-subnet-mgr],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:FLANNELD_IFACE,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:rke2-canal-config,},Key:canal_iface,Optional:nil,},SecretKeyRef:nil,},},EnvVar{Name:FLANNELD_IFACE_REGEX,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:rke2-canal-config,},Key:canal_iface_regex,Optional:nil,},SecretKeyRef:nil,},},EnvVar{Name:FLANNELD_IP_MASQ,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:rke2-canal-config,},Key:masquerade,Optional:nil,},SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:flannel-cfg,ReadOnly:false,MountPath:/etc/kube-flannel/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-qrxdq,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars
E0908 21:11:24.174319    9799 pod_workers.go:1300] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"calico-node\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\", failed to \"StartContainer\" for \"kube-flannel\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"]" pod="kube-system/rke2-canal-4nblx" podUID="3a400dab-6eb5-47f3-b87d-2867f168bd7f"
I0908 21:11:24.203808    9799 status_manager.go:874] "Patch status for pod" pod="kube-system/rke2-canal-4nblx" podUID="3a400dab-6eb5-47f3-b87d-2867f168bd7f" patch="{\"metadata\":{\"uid\":\"3a400dab-6eb5-47f3-b87d-2867f168bd7f\"},\"status\":{\"initContainerStatuses\":[{\"containerID\":\"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c\",\"image\":\"docker.io/rancher/hardened-calico:v3.26.1-build20230802\",\"imageID\":\"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c\",\"lastState\":{\"terminated\":{\"containerID\":\"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c\",\"exitCode\":0,\"finishedAt\":\"2023-09-08T21:07:49Z\",\"reason\":\"Completed\",\"startedAt\":\"2023-09-08T21:07:46Z\"}},\"name\":\"install-cni\",\"ready\":true,\"restartCount\":0,\"started\":false,\"state\":{\"waiting\":{\"message\":\"services have not yet been read at least once, cannot construct envvars\",\"reason\":\"CreateContainerConfigError\"}}},{\"containerID\":\"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b\",\"image\":\"docker.io/rancher/hardened-calico:v3.26.1-build20230802\",\"imageID\":\"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c\",\"lastState\":{},\"name\":\"flexvol-driver\",\"ready\":true,\"restartCount\":0,\"started\":false,\"state\":{\"terminated\":{\"containerID\":\"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b\",\"exitCode\":0,\"finishedAt\":\"2023-09-08T21:07:51Z\",\"reason\":\"Completed\",\"startedAt\":\"2023-09-08T21:07:51Z\"}}}]}}"
I0908 21:11:24.203890    9799 status_manager.go:883] "Status for pod updated successfully" pod="kube-system/rke2-canal-4nblx" statusVersion=2 status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:52Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e","started":false},{"name":"kube-flannel","state":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:54Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7","started":false}],"qosClass":"Burstable"}
E0908 21:11:25.174696    9799 kuberuntime_manager.go:1209] container &Container{Name:calico-node,Image:rancher/hardened-calico:v3.26.1-build20230802,Command:[start_runit],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:DATASTORE_TYPE,Value:kubernetes,ValueFrom:nil,},EnvVar{Name:USE_POD_CIDR,Value:true,ValueFrom:nil,},EnvVar{Name:WAIT_FOR_DATASTORE,Value:true,ValueFrom:nil,},EnvVar{Name:NODENAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:CALICO_CNI_SERVICE_ACCOUNT,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.serviceAccountName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:CALICO_NETWORKING_BACKEND,Value:none,ValueFrom:nil,},EnvVar{Name:CLUSTER_TYPE,Value:k8s,canal,ValueFrom:nil,},EnvVar{Name:FELIX_IPTABLESREFRESHINTERVAL,Value:60,ValueFrom:nil,},EnvVar{Name:FELIX_IPTABLESBACKEND,Value:auto,ValueFrom:nil,},EnvVar{Name:CALICO_DISABLE_FILE_LOGGING,Value:true,ValueFrom:nil,},EnvVar{Name:FELIX_DEFAULTENDPOINTTOHOSTACTION,Value:ACCEPT,ValueFrom:nil,},EnvVar{Name:FELIX_IPV6SUPPORT,Value:false,ValueFrom:nil,},EnvVar{Name:FELIX_LOGSEVERITYSCREEN,Value:info,ValueFrom:nil,},EnvVar{Name:FELIX_HEALTHENABLED,Value:true,ValueFrom:nil,},EnvVar{Name:FELIX_PROMETHEUSMETRICSENABLED,Value:true,ValueFrom:nil,},EnvVar{Name:FELIX_XDPENABLED,Value:false,ValueFrom:nil,},EnvVar{Name:FELIX_FAILSAFEINBOUNDHOSTPORTS,Value:,ValueFrom:nil,},EnvVar{Name:FELIX_FAILSAFEOUTBOUNDHOSTPORTS,Value:,ValueFrom:nil,},EnvVar{Name:IP_AUTODETECTION_METHOD,Value:first-found,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{250 -3} {<nil>} 250m DecimalSI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:cni-net-dir,ReadOnly:false,MountPath:/host/etc/cni/net.d,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:lib-modules,ReadOnly:true,MountPath:/lib/modules,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:var-run-calico,ReadOnly:false,MountPath:/var/run/calico,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:var-lib-calico,ReadOnly:false,MountPath:/var/lib/calico,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:policysync,ReadOnly:false,MountPath:/var/run/nodeagent,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-log-dir,ReadOnly:true,MountPath:/var/log/calico/cni,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-qrxdq,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[/usr/bin/calico-node -felix-live],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:10,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:6,TerminationGracePeriodSeconds:nil,},ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readiness,Port:{0 9099 },Host:localhost,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/usr/bin/calico-node -shutdown],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars
E0908 21:11:25.175721    9799 kuberuntime_manager.go:1209] container &Container{Name:kube-flannel,Image:rancher/hardened-flannel:v0.22.1-build20230802,Command:[/opt/bin/flanneld --ip-masq --kube-subnet-mgr],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:FLANNELD_IFACE,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:rke2-canal-config,},Key:canal_iface,Optional:nil,},SecretKeyRef:nil,},},EnvVar{Name:FLANNELD_IFACE_REGEX,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:rke2-canal-config,},Key:canal_iface_regex,Optional:nil,},SecretKeyRef:nil,},},EnvVar{Name:FLANNELD_IP_MASQ,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:rke2-canal-config,},Key:masquerade,Optional:nil,},SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:flannel-cfg,ReadOnly:false,MountPath:/etc/kube-flannel/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-qrxdq,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars
E0908 21:11:25.175783    9799 pod_workers.go:1300] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"calico-node\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\", failed to \"StartContainer\" for \"kube-flannel\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"]" pod="kube-system/rke2-canal-4nblx" podUID="3a400dab-6eb5-47f3-b87d-2867f168bd7f"
I0908 21:11:25.186299    9799 status_manager.go:874] "Patch status for pod" pod="kube-system/rke2-canal-4nblx" podUID="3a400dab-6eb5-47f3-b87d-2867f168bd7f" patch="{\"metadata\":{\"uid\":\"3a400dab-6eb5-47f3-b87d-2867f168bd7f\"},\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastTransitionTime\":\"2023-09-08T21:11:25Z\",\"message\":\"containers with incomplete status: [install-cni]\",\"reason\":\"ContainersNotInitialized\",\"status\":\"False\",\"type\":\"Initialized\"}],\"containerStatuses\":[{\"containerID\":\"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e\",\"image\":\"docker.io/rancher/hardened-calico:v3.26.1-build20230802\",\"imageID\":\"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c\",\"lastState\":{\"terminated\":{\"containerID\":\"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e\",\"exitCode\":255,\"finishedAt\":\"2023-09-08T21:09:15Z\",\"reason\":\"Unknown\",\"startedAt\":\"2023-09-08T21:07:52Z\"}},\"name\":\"calico-node\",\"ready\":false,\"restartCount\":0,\"started\":false,\"state\":{\"waiting\":{\"message\":\"services have not yet been read at least once, cannot construct envvars\",\"reason\":\"CreateContainerConfigError\"}}},{\"containerID\":\"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7\",\"image\":\"docker.io/rancher/hardened-flannel:v0.22.1-build20230802\",\"imageID\":\"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439\",\"lastState\":{\"terminated\":{\"containerID\":\"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7\",\"exitCode\":255,\"finishedAt\":\"2023-09-08T21:09:15Z\",\"reason\":\"Unknown\",\"startedAt\":\"2023-09-08T21:07:54Z\"}},\"name\":\"kube-flannel\",\"ready\":false,\"restartCount\":0,\"started\":false,\"state\":{\"waiting\":{\"message\":\"services have not yet been read at least once, cannot construct envvars\",\"reason\":\"CreateContainerConfigError\"}}}],\"initContainerStatuses\":[{\"image\":\"rancher/hardened-calico:v3.26.1-build20230802\",\"imageID\":\"\",\"lastState\":{\"terminated\":{\"containerID\":\"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c\",\"exitCode\":0,\"finishedAt\":\"2023-09-08T21:07:49Z\",\"reason\":\"Completed\",\"startedAt\":\"2023-09-08T21:07:46Z\"}},\"name\":\"install-cni\",\"ready\":false,\"restartCount\":0,\"started\":false,\"state\":{\"waiting\":{\"message\":\"services have not yet been read at least once, cannot construct envvars\",\"reason\":\"CreateContainerConfigError\"}}},{\"containerID\":\"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b\",\"image\":\"docker.io/rancher/hardened-calico:v3.26.1-build20230802\",\"imageID\":\"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c\",\"lastState\":{},\"name\":\"flexvol-driver\",\"ready\":true,\"restartCount\":0,\"started\":false,\"state\":{\"terminated\":{\"containerID\":\"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b\",\"exitCode\":0,\"finishedAt\":\"2023-09-08T21:07:51Z\",\"reason\":\"Completed\",\"startedAt\":\"2023-09-08T21:07:51Z\"}}}]}}"
I0908 21:11:25.186389    9799 status_manager.go:883] "Status for pod updated successfully" pod="kube-system/rke2-canal-4nblx" statusVersion=3 status={"phase":"Running","conditions":[{"type":"Initialized","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:25Z","reason":"ContainersNotInitialized","message":"containers with incomplete status: [install-cni]"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":false,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e","started":false},{"name":"kube-flannel","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:54Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7"}},"ready":false,"restartCount":0,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7","started":false}],"qosClass":"Burstable"}
I0908 21:11:38.073156    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:25Z","reason":"ContainersNotInitialized","message":"containers with incomplete status: [install-cni]"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":false,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e","started":false},{"name":"kube-flannel","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:54Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7"}},"ready":false,"restartCount":0,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7","started":false}],"qosClass":"Burstable"}
I0908 21:11:39.223606    9799 status_manager.go:883] "Status for pod updated successfully" pod="kube-system/rke2-canal-4nblx" statusVersion=4 status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:54Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8","started":true}],"qosClass":"Burstable"}
I0908 21:11:40.210392    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:54Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8","started":true}],"qosClass":"Burstable"}
I0908 21:12:09.299170    9799 status_manager.go:883] "Status for pod updated successfully" pod="kube-system/rke2-canal-4nblx" statusVersion=5 status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:54Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://546bff185f7be96ed4e32753038b9acca476e8237051547fe626deff011b78e7"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8","started":false}],"qosClass":"Burstable"}
I0908 21:12:23.084640    9799 status_manager.go:883] "Status for pod updated successfully" pod="kube-system/rke2-canal-4nblx" statusVersion=6 status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"waiting":{"reason":"CrashLoopBackOff","message":"back-off 40s restarting failed container=kube-flannel pod=rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f)"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8","started":false}],"qosClass":"Burstable"}
I0908 21:12:37.072514    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [calico-node kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"waiting":{"reason":"CrashLoopBackOff","message":"back-off 40s restarting failed container=kube-flannel pod=rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f)"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8","started":false}],"qosClass":"Burstable"}
I0908 21:12:43.362697    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"waiting":{"reason":"CrashLoopBackOff","message":"back-off 40s restarting failed container=kube-flannel pod=rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f)"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8","started":false}],"qosClass":"Burstable"}
I0908 21:12:43.376727    9799 status_manager.go:883] "Status for pod updated successfully" pod="kube-system/rke2-canal-4nblx" statusVersion=7 status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"waiting":{"reason":"CrashLoopBackOff","message":"back-off 40s restarting failed container=kube-flannel pod=rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f)"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8","started":false}],"qosClass":"Burstable"}
I0908 21:12:58.072697    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [kube-flannel]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:23Z","reason":"ContainersNotReady","message":"containers with unready status: [kube-flannel]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"waiting":{"reason":"CrashLoopBackOff","message":"back-off 40s restarting failed container=kube-flannel pod=rke2-canal-4nblx_kube-system(3a400dab-6eb5-47f3-b87d-2867f168bd7f)"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":false,"restartCount":1,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8","started":false}],"qosClass":"Burstable"}
I0908 21:12:58.450735    9799 status_manager.go:883] "Status for pod updated successfully" pod="kube-system/rke2-canal-4nblx" statusVersion=8 status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"running":{"startedAt":"2023-09-08T21:12:58Z"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":true,"restartCount":2,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://22b52ccb4ea17b2f2c97637d1beaa1b7b84fab8d4f4beb6fec2a01d066e61a8d","started":true}],"qosClass":"Burstable"}
I0908 21:13:34.071751    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"running":{"startedAt":"2023-09-08T21:12:58Z"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":true,"restartCount":2,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://22b52ccb4ea17b2f2c97637d1beaa1b7b84fab8d4f4beb6fec2a01d066e61a8d","started":true}],"qosClass":"Burstable"}
I0908 21:14:11.072175    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"running":{"startedAt":"2023-09-08T21:12:58Z"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":true,"restartCount":2,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://22b52ccb4ea17b2f2c97637d1beaa1b7b84fab8d4f4beb6fec2a01d066e61a8d","started":true}],"qosClass":"Burstable"}
I0908 21:14:49.072611    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"running":{"startedAt":"2023-09-08T21:12:58Z"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":true,"restartCount":2,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://22b52ccb4ea17b2f2c97637d1beaa1b7b84fab8d4f4beb6fec2a01d066e61a8d","started":true}],"qosClass":"Burstable"}
I0908 21:15:30.072260    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"running":{"startedAt":"2023-09-08T21:12:58Z"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":true,"restartCount":2,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://22b52ccb4ea17b2f2c97637d1beaa1b7b84fab8d4f4beb6fec2a01d066e61a8d","started":true}],"qosClass":"Burstable"}
I0908 21:16:03.072428    9799 status_manager.go:693] "Ignoring same status for pod" pod="kube-system/rke2-canal-4nblx" status={"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:11:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:12:58Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-09-08T21:07:39Z"}],"hostIP":"172.17.0.3","podIP":"172.17.0.3","podIPs":[{"ip":"172.17.0.3"},{"ip":"fd7c:53a5:aef5::242:ac11:3"}],"startTime":"2023-09-08T21:07:39Z","initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:46Z","finishedAt":"2023-09-08T21:07:49Z","containerID":"containerd://726d8df8320c5c1bde766b64209010accb93ca3c4ecad9b1593f470c71f94c3c"}},"ready":false,"restartCount":0,"image":"rancher/hardened-calico:v3.26.1-build20230802","imageID":"","started":false},{"name":"flexvol-driver","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2023-09-08T21:07:51Z","finishedAt":"2023-09-08T21:07:51Z","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://2e7795576805f5d7be7954b9f2d0e7b0f6454a0a7445d56b27ebe2bc7eaf9c0b","started":false}],"containerStatuses":[{"name":"calico-node","state":{"running":{"startedAt":"2023-09-08T21:11:38Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Unknown","startedAt":"2023-09-08T21:07:52Z","finishedAt":"2023-09-08T21:09:15Z","containerID":"containerd://bb68d989c5254c4fc4e1ecdc262b087369b4121690de2f5503287944f95e136e"}},"ready":true,"restartCount":1,"image":"docker.io/rancher/hardened-calico:v3.26.1-build20230802","imageID":"docker.io/rancher/hardened-calico@sha256:b7f61a9bcc6c144d76f489b446e2eb0fcac8a751d2ca4de838af07e112b2545c","containerID":"containerd://6a0073f0bf84751433fa3ed87b400fc56f7e12d0392b64cf84eefa5888d9f095","started":true},{"name":"kube-flannel","state":{"running":{"startedAt":"2023-09-08T21:12:58Z"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2023-09-08T21:11:38Z","finishedAt":"2023-09-08T21:12:08Z","containerID":"containerd://e4db6d06246ce820b9271a5894f91debac5b3ae40a76e616aa873715e035b9d8"}},"ready":true,"restartCount":2,"image":"docker.io/rancher/hardened-flannel:v0.22.1-build20230802","imageID":"docker.io/rancher/hardened-flannel@sha256:fc31bf0b99a9727f37e6a222c21f85d016dee13869a6c7ca26d82f4395371439","containerID":"containerd://22b52ccb4ea17b2f2c97637d1beaa1b7b84fab8d4f4beb6fec2a01d066e61a8d","started":true}],"qosClass":"Burstable"}

"initContainerStatuses":[{"name":"install-cni","state":{"waiting":{"reason":"CreateContainerConfigError","message":"services have not yet been read at least once, cannot construct envvars"}},

This appears to be a change in behavior (bug perhaps) in the 1.28 kubelet.

I do see that it has successfully listed services, but for some reason it's not retrying the pod setup.

I0908 21:11:26.148028    9799 reflector.go:325] Listing and watching *v1.Service from k8s.io/client-go/informers/factory.go:150
I0908 21:11:26.149310    9799 round_trippers.go:553] GET https://127.0.0.1:6443/api/v1/services?limit=500&resourceVersion=0 200 OK in 1 milliseconds

@brandond
Copy link
Member

brandond commented Sep 8, 2023

@brandond
Copy link
Member

brandond commented Sep 9, 2023

Closing as upstream issue.

@brandond brandond closed this as not planned Won't fix, can't repro, duplicate, stale Sep 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@manuelbuil manuelbuil

@rbrtbnfgl rbrtbnfgl

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@brandond @manuelbuil @rbrtbnfgl @ShylajaDevadiga @est-suse