Dual-stack not working #3757

proea · 2022-12-07T21:02:07Z

proea
Dec 7, 2022

Environmental Info:
RKE2 Version:

rke2 version v1.24.8+rke2r1 (b061438ce1f04518d980aac842b62f92095d793d)
go version go1.18.1b7

Node(s) CPU architecture, OS, and Version:
Linux <redacted> 4.18.0-425.3.1.el8.x86_64 #1 SMP Wed Nov 9 20:13:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
One node
Describe the bug:
Pods are failing to start

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "42320ff5e29dd42b1df8521a361632a300449eb2f0697bae417133ca32b34a16": plugin type="calico" failed (add): usePodCidrIPv6 found but there is no IPv6 CIDR configured

Steps To Reproduce:

Installed RKE2
Changed /etc/rancher/rke2/config.yaml

node-ip: <redacted-ipv4>,<redacted-ipv6>
cluster-cidr: 10.42.0.0/16,2001:cafe:42:0::/56
service-cidr: 10.43.0.0/16,2001:cafe:42:1::/112
cluster-dns: 10.43.0.10

Restarted RKE2 systemctl restart rke2-server.service

Expected behavior:
ipv6 is available. Pods have ipv6 address. Pods can reach global network using ipv6

Actual behavior:
IPv6 network doesnt work. System pods are failed to start

Additional context / logs:
rke2-canal-hq7hp also has error

Readiness probe failed: Get "http://localhost:9099/readiness": dial tcp [::1]:9099: connect: connection refused
Back-off restarting failed container

Click me

...
2022-12-07 20:50:44.709 [INFO][53] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"} ipVersion=0x6 status="up"
2022-12-07 20:50:44.709 [INFO][53] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"} status="up"
2022-12-07 20:51:15.731 [WARNING][50] monitor-addresses/startup.go 621: Unable to confirm IPv4 address 172.18.0.1 is assigned to this host
2022-12-07 20:51:23.912 [INFO][53] felix/summary.go 100: Summarising 14 dataplane reconciliation loops over 1m0s: avg=11ms longest=40ms (resync-filter-v4,resync-filter-v6)
2022-12-07 20:51:44.856 [INFO][53] felix/calc_graph.go 462: Local endpoint updated id=WorkloadEndpoint(node=<redacted-host>, orchestrator=k8s, workload=awx/awx-service-5c98d56878-t4z98, name=eth0)
2022-12-07 20:51:44.857 [INFO][53] felix/int_dataplane.go 1680: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"awx/awx-service-5c98d56878-t4z98" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali44804531a13" profile_ids:"kns.awx" profile_ids:"ksa.awx.awx-service" ipv4_nets:"10.42.0.17/32" > 
2022-12-07 20:51:44.857 [INFO][53] felix/endpoint_mgr.go 600: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:44.857 [INFO][53] felix/table.go 508: Queueing update of chain. chainName="cali-tw-cali44804531a13" ipVersion=0x4 table="filter"
2022-12-07 20:51:44.857 [INFO][53] felix/table.go 508: Queueing update of chain. chainName="cali-fw-cali44804531a13" ipVersion=0x4 table="filter"
2022-12-07 20:51:44.857 [INFO][53] felix/endpoint_mgr.go 646: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:44.858 [INFO][53] felix/endpoint_mgr.go 1212: Applying /proc/sys configuration to interface. ifaceName="cali44804531a13"
2022-12-07 20:51:44.858 [INFO][53] felix/endpoint_mgr.go 488: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:44.858 [INFO][53] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:44.858 [INFO][53] felix/endpoint_mgr.go 600: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:44.858 [INFO][53] felix/table.go 508: Queueing update of chain. chainName="cali-tw-cali44804531a13" ipVersion=0x6 table="filter"
2022-12-07 20:51:44.858 [INFO][53] felix/table.go 508: Queueing update of chain. chainName="cali-fw-cali44804531a13" ipVersion=0x6 table="filter"
2022-12-07 20:51:44.858 [INFO][53] felix/endpoint_mgr.go 646: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:44.859 [INFO][53] felix/endpoint_mgr.go 1212: Applying /proc/sys configuration to interface. ifaceName="cali44804531a13"
2022-12-07 20:51:44.859 [INFO][53] felix/endpoint_mgr.go 488: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:44.859 [INFO][53] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x6 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:44.895 [INFO][53] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"} ipVersion=0x4 status="up"
2022-12-07 20:51:44.895 [INFO][53] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"} status="up"
2022-12-07 20:51:45.854 [INFO][53] felix/calc_graph.go 462: Local endpoint updated id=WorkloadEndpoint(node=<redacted-host>, orchestrator=k8s, workload=awx/awx-service-5c98d56878-t4z98, name=eth0)
2022-12-07 20:51:45.854 [INFO][53] felix/int_dataplane.go 1680: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"awx/awx-service-5c98d56878-t4z98" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali44804531a13" profile_ids:"kns.awx" profile_ids:"ksa.awx.awx-service" ipv4_nets:"10.42.0.17/32" > 
2022-12-07 20:51:45.854 [INFO][53] felix/endpoint_mgr.go 600: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:45.854 [INFO][53] felix/table.go 508: Queueing update of chain. chainName="cali-tw-cali44804531a13" ipVersion=0x4 table="filter"
2022-12-07 20:51:45.854 [INFO][53] felix/table.go 508: Queueing update of chain. chainName="cali-fw-cali44804531a13" ipVersion=0x4 table="filter"
2022-12-07 20:51:45.854 [INFO][53] felix/endpoint_mgr.go 646: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:45.855 [INFO][53] felix/endpoint_mgr.go 1212: Applying /proc/sys configuration to interface. ifaceName="cali44804531a13"
2022-12-07 20:51:45.855 [INFO][53] felix/endpoint_mgr.go 488: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:45.855 [INFO][53] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:45.855 [INFO][53] felix/endpoint_mgr.go 600: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:45.855 [INFO][53] felix/table.go 508: Queueing update of chain. chainName="cali-tw-cali44804531a13" ipVersion=0x6 table="filter"
2022-12-07 20:51:45.855 [INFO][53] felix/table.go 508: Queueing update of chain. chainName="cali-fw-cali44804531a13" ipVersion=0x6 table="filter"
2022-12-07 20:51:45.855 [INFO][53] felix/endpoint_mgr.go 646: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:45.856 [INFO][53] felix/endpoint_mgr.go 1212: Applying /proc/sys configuration to interface. ifaceName="cali44804531a13"
2022-12-07 20:51:45.856 [INFO][53] felix/endpoint_mgr.go 488: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:45.856 [INFO][53] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x6 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"}
2022-12-07 20:51:45.884 [INFO][53] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"} ipVersion=0x4 status="up"
2022-12-07 20:51:45.884 [INFO][53] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"awx/awx-service-5c98d56878-t4z98", EndpointId:"eth0"} status="up"
2022-12-07 20:52:15.733 [WARNING][50] monitor-addresses/startup.go 621: Unable to confirm IPv4 address 172.18.0.1 is assigned to this host
2022-12-07 20:52:23.914 [INFO][53] felix/summary.go 100: Summarising 15 dataplane reconciliation loops over 1m0s: avg=9ms longest=38ms (resync-filter-v4,resync-filter-v6)
2022-12-07 20:53:15.735 [WARNING][50] monitor-addresses/startup.go 621: Unable to confirm IPv4 address 172.18.0.1 is assigned to this host
2022-12-07 20:53:23.916 [INFO][53] felix/summary.go 100: Summarising 13 dataplane reconciliation loops over 1m0s: avg=6ms longest=14ms (resync-routes-v4,resync-routes-v4,resync-routes-v6,resync-routes-v6,resync-rules-v4,resync-rules-v6,resync-wg,resync-wg)
2022-12-07 20:54:15.737 [WARNING][50] monitor-addresses/startup.go 621: Unable to confirm IPv4 address 172.18.0.1 is assigned to this host
2022-12-07 20:54:23.918 [INFO][53] felix/summary.go 100: Summarising 13 dataplane reconciliation loops over 1m0s: avg=6ms longest=14ms (resync-filter-v4,resync-filter-v6)

kubectl get nodes -o yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      csi.volume.kubernetes.io/nodeid: '{"driver.longhorn.io":"<redacted-host>"}'
      etcd.rke2.cattle.io/node-address: <redacted-ipv4>
      etcd.rke2.cattle.io/node-name: <redacted-host>-5ab7096b
      flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"ca:ab:00:2b:37:c6"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/backend-v6-data: '{"VNI":1,"VtepMAC":"c6:d9:61:fb:0d:68"}'
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: <redacted-ipv4>
      flannel.alpha.coreos.com/public-ipv6: <redacted-ipv6>
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 172.18.0.1/16
      projectcalico.org/IPv4IPIPTunnelAddr: 10.42.0.1
      rke2.io/encryption-config-hash: start-c8bef0656005808a1ef0b36fc652ac94589e2bef20ddfbea013ce923aa27021f
      rke2.io/hostname: <redacted-host>
      rke2.io/internal-ip: <redacted-ipv4>,<redacted-ipv6>
      rke2.io/node-args: '["server","--node-ip","<redacted-ipv4>,<redacted-ipv6>","--cluster-cidr","10.42.0.0/16,2001:cafe:42:0::/56","--service-cidr","10.43.0.0/16,2001:cafe:42:1::/112","--cluster-dns","10.43.0.10"]'
      rke2.io/node-config-hash: ...
      rke2.io/node-env: '{"RKE2_SELINUX":"true"}'
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2022-12-07T16:49:34Z"
    finalizers:
    - wrangler.cattle.io/node
    - wrangler.cattle.io/managed-etcd-controller
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: rke2
      beta.kubernetes.io/os: linux
      egress.rke2.io/cluster: "true"
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: <redacted-host>
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: "true"
      node-role.kubernetes.io/etcd: "true"
      node-role.kubernetes.io/master: "true"
      node.kubernetes.io/instance-type: rke2
    name: <redacted-host>
    resourceVersion: "68801"
    uid: 0cf25aba-f6b3-4031-a4e2-90c831f5c133
  spec:
    podCIDR: 10.42.0.0/24
    podCIDRs:
    - 10.42.0.0/24
    providerID: rke2://<redacted-host>
  status:
    addresses:
    - address: <redacted-ipv4>
      type: InternalIP
    - address: <redacted-ipv6>
      type: InternalIP
    - address: <redacted-host>
      type: Hostname
    allocatable:
      cpu: "8"
      ephemeral-storage: "35716344190"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7948688Ki
      pods: "110"
    capacity:
      cpu: "8"
      ephemeral-storage: 36714992Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7948688Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2022-12-07T20:06:42Z"
      lastTransitionTime: "2022-12-07T20:06:42Z"
      message: Flannel is running on this node
      reason: FlannelIsUp
      status: "False"
      type: NetworkUnavailable
    - lastHeartbeatTime: "2022-12-07T20:53:25Z"
      lastTransitionTime: "2022-12-07T16:49:34Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2022-12-07T20:53:25Z"
      lastTransitionTime: "2022-12-07T16:49:34Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2022-12-07T20:53:25Z"
      lastTransitionTime: "2022-12-07T16:49:34Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2022-12-07T20:53:25Z"
      lastTransitionTime: "2022-12-07T20:46:57Z"
      message: kubelet is posting ready status
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - quay.io/ansible/awx-ee@sha256:30d59eff5bafa91a79d2034dc30ba1ba17c5ae94ec033c3398710232393b89d5
      - quay.io/ansible/awx-ee:latest
      sizeBytes: 642557584
    - names:
      - docker.io/rancher/nginx-ingress-controller@sha256:6be4f6e2c0f4a6fdce39b3c225ef5ad864e8fd0990ce26260719dee3aafcfcf6
      - docker.io/rancher/nginx-ingress-controller:nginx-1.2.1-hardened9
      sizeBytes: 315888001
    - names:
      - docker.io/longhornio/longhorn-engine@sha256:1714ebca90b22aa6aca9f8169cb5619e9df851f3a2981411efdb8853f0d31e61
      - docker.io/longhornio/longhorn-engine:v1.3.2
      sizeBytes: 285906627
    - names:
      - docker.io/longhornio/longhorn-instance-manager@sha256:924210972c43f260485482b41ca9d768fe3a3ed4630f5091371e517fa361b0cf
      - docker.io/longhornio/longhorn-instance-manager:v1_20221003
      sizeBytes: 282443929
    - names:
      - quay.io/ansible/awx@sha256:ac4509fdc5e9cb547ee9a95a00e0941d890b8598e89f138ae8c834ad59555d64
      - quay.io/ansible/awx:21.10.0
      sizeBytes: 220326941
    - names:
      - docker.io/rancher/hardened-kubernetes@sha256:c9b4b723041f09938d807f633b07d69c125ce40430d3e8c5c5fc8c4e777822e3
      - docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110
      sizeBytes: 210081875
    - names:
      - docker.io/rancher/hardened-calico@sha256:844c5e91732879b91a022c4000ac3ada05e7e7d91e3c163d0bb4b8488d48c303
      - docker.io/rancher/hardened-calico:v3.24.1-build20221011
      sizeBytes: 207545413
    - names:
      - quay.io/ansible/awx-operator@sha256:f95764256cf1c368d104d740f37ae44cd94813b5fd28c303f7f7c64a284a0ab5
      - quay.io/ansible/awx-operator:1.1.1
      sizeBytes: 179899342
    - names:
      - docker.io/library/postgres@sha256:5fec4106f03419cb92dd604a8dd2ae85e724c640af743ba3d24ea2198f762250
      - docker.io/library/postgres:13
      sizeBytes: 136692140
    - names:
      - docker.io/longhornio/longhorn-manager@sha256:3e43951818585a7dad7544f594aff15e9f566980c95c933251ccb1d2b76a419b
      - docker.io/longhornio/longhorn-manager:v1.3.2
      sizeBytes: 101056614
    - names:
      - docker.io/rancher/klipper-helm@sha256:6a8e819402e3fdd5ff9ec576174b6c0013870b9c0627a05fa0ab17374b5cf189
      - docker.io/rancher/klipper-helm:v0.7.3-build20220613
      sizeBytes: 82983731
    - names:
      - docker.io/rancher/hardened-flannel@sha256:f8b624655da659bcbcc035383fd6bada12c59d4921683cf1a7167da525f3b53b
      - docker.io/rancher/hardened-flannel:v0.19.1-build20221011
      sizeBytes: 81706270
    - names:
      - docker.io/kubernetesui/dashboard@sha256:290bebc3cd96c22b6f89e7b21f5c2b16ce5c275a0ec2c2de10e0d8b9dd110289
      - docker.io/kubernetesui/dashboard:v2.6.1
      sizeBytes: 75789999
    - names:
      - docker.io/kubernetesui/dashboard@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93
      - docker.io/kubernetesui/dashboard:v2.7.0
      sizeBytes: 75788960
    - names:
      - docker.io/longhornio/longhorn-ui@sha256:cd601af395504997dd44dfd04ee1e94797c68c34d184a5a4388f23e7ba8ba6af
      - docker.io/longhornio/longhorn-ui:v1.3.2
      sizeBytes: 72812679
    - names:
      - docker.io/rancher/hardened-etcd@sha256:abb0674e4f1462153f3542ea90c0272988e024d304fe9e1f3267643d744caaab
      - docker.io/rancher/hardened-etcd:v3.5.4-k3s1-build20221011
      sizeBytes: 64814391
    - names:
      - docker.io/rancher/hardened-coredns@sha256:3020505d755499eb8d22f631aa88fc597b4849eae18404401087669a6bc5ca06
      - docker.io/rancher/hardened-coredns:v1.9.3-build20221011
      sizeBytes: 64087613
    - names:
      - docker.io/rancher/rke2-cloud-provider@sha256:6d8417bd79a7f2a61fc0b3b8801520baf12861e79c1027994b26590522edf2d9
      - docker.io/rancher/rke2-cloud-provider:v1.25.3-build20221017
      sizeBytes: 63467155
    - names:
      - docker.io/rancher/hardened-k8s-metrics-server@sha256:2d970ba32946f6c2b5f46cae4f5219d4539d4310ac0a4542ca13e4233bd8f4a8
      - docker.io/rancher/hardened-k8s-metrics-server:v0.6.1-build20221011
      sizeBytes: 63382334
    - names:
      - docker.io/rancher/hardened-cluster-autoscaler@sha256:18cf5a3f1c2c7df073b310016b1ac28e8e543afb5cffe91da6bc33176f0fb9f8
      - docker.io/rancher/hardened-cluster-autoscaler:v1.8.5-build20221011
      sizeBytes: 56358657
    - names:
      - docker.io/library/redis@sha256:dfeb5451fce377ab47c5bb6b6826592eea534279354bbfc3890c0b5e9b57c763
      - docker.io/library/redis:7
      sizeBytes: 42400032
    - names:
      - gcr.io/kubebuilder/kube-rbac-proxy@sha256:d99a8d144816b951a67648c12c0b988936ccd25cf3754f3cd85ab8c01592248f
      - gcr.io/kubebuilder/kube-rbac-proxy:v0.13.0
      sizeBytes: 25405613
    - names:
      - docker.io/longhornio/csi-attacher@sha256:365b6e745404cf3eb39ee9bcc55b1af3c4f5357e44dddd0270c3cbb06b252a64
      - docker.io/longhornio/csi-attacher:v3.4.0
      sizeBytes: 22084988
    - names:
      - docker.io/longhornio/csi-resizer@sha256:76c3623cb51b8bc0de7588812e0f684221d3db3ffd739564e92a1cb68892ac70
      - docker.io/longhornio/csi-resizer:v1.2.0
      sizeBytes: 21584301
    - names:
      - docker.io/longhornio/csi-provisioner@sha256:daa7828fd45897fdeec17d6e987bc54cdc0a0928a7ba6b62d8f8496d6aa5d3be
      - docker.io/longhornio/csi-provisioner:v2.1.2
      sizeBytes: 21220203
    - names:
      - docker.io/kubernetesui/metrics-scraper@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c
      - docker.io/kubernetesui/metrics-scraper:v1.0.8
      sizeBytes: 19746404
    - names:
      - docker.io/longhornio/csi-snapshotter@sha256:fb4b91b90e4017895ed07cf755a8330256dac92bd52c64dc05c45c4f0b740506
      - docker.io/longhornio/csi-snapshotter:v3.0.3
      sizeBytes: 19500467
    - names:
      - docker.io/rancher/mirrored-ingress-nginx-kube-webhook-certgen@sha256:28197903d736aae74cbb1fa9e0ccbd11129395f0f65ad94281cc7fdfec020b25
      - docker.io/rancher/mirrored-ingress-nginx-kube-webhook-certgen:v1.1.1
      sizeBytes: 18899182
    - names:
      - quay.io/jetstack/cert-manager-controller@sha256:1143471c90db621faee43ed53f250d0fcec9c86303ca83661c787d59486d2ff4
      - quay.io/jetstack/cert-manager-controller:v1.10.1
      sizeBytes: 17740776
    - names:
      - quay.io/jetstack/cert-manager-webhook@sha256:64121721c6659e6790d7de527e8bc115ffc28b6e73e1b9a7f9aa11a90286fe25
      - quay.io/jetstack/cert-manager-webhook:v1.10.1
      sizeBytes: 13727551
    - names:
      - quay.io/jetstack/cert-manager-cainjector@sha256:b5657161d2c2f74ab292da4cbaba3923bc74112525f70c3d884fb3c6ef83c303
      - quay.io/jetstack/cert-manager-cainjector:v1.10.1
      sizeBytes: 12201630
    - names:
      - docker.io/longhornio/csi-node-driver-registrar@sha256:b3288cdcb832c30acc90f0d6363fc681880f46448f0175078bbd07f2a178634e
      - docker.io/longhornio/csi-node-driver-registrar:v2.5.0
      sizeBytes: 9132327
    - names:
      - docker.io/library/alpine@sha256:c75ac27b49326926b803b9ed43bf088bc220d22556de1bc5f72d742c91398f69
      - docker.io/library/alpine:3.12
      sizeBytes: 2811698
    - names:
      - docker.io/rancher/pause@sha256:036d575e82945c112ef84e4585caff3648322a2f9ed4c3a6ce409dd10abc4f34
      - docker.io/rancher/pause:3.6
      sizeBytes: 299396
    nodeInfo:
      architecture: amd64
      bootID: 20908e6d-9358-4696-a090-c0e76e6b94af
      containerRuntimeVersion: containerd://1.6.8-k3s1
      kernelVersion: 4.18.0-425.3.1.el8.x86_64
      kubeProxyVersion: v1.24.8+rke2r1
      kubeletVersion: v1.24.8+rke2r1
      machineID: 872c8375b1d44b209a6ed1c5504d2ac5
      operatingSystem: linux
      osImage: Rocky Linux 8.7 (Green Obsidian)
      systemUUID: 230062d8-b2a9-4085-a00f-6e069d466870
    volumesAttached:
    - devicePath: ""
      name: kubernetes.io/csi/driver.longhorn.io^pvc-b23aa16c-3bd5-49a5-ad41-f69891589cdb
    volumesInUse:
    - kubernetes.io/csi/driver.longhorn.io^pvc-b23aa16c-3bd5-49a5-ad41-f69891589cdb
kind: List
metadata:
  resourceVersion: ""

Answered by brandond

Dec 8, 2022

CNI selection and cluster/service CIDRs cannot be changed once the cluster has already been started. We can leave this open to track adding this to the docs if you'd like.

View full answer

proea · 2022-12-07T21:16:28Z

proea
Dec 7, 2022
Author

kubectl describe -n kube-system pod/kube-controller-manager-$(hostname)

Name:                 kube-controller-manager-<redacted-host>
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 <redacted-host>/<redacted-ipv4>
Start Time:           Wed, 07 Dec 2022 22:05:06 +0100
Labels:               component=kube-controller-manager
                      tier=control-plane
Annotations:          kubernetes.io/config.hash: 57585a0305e4e46df816ebab263926f3
                      kubernetes.io/config.mirror: 57585a0305e4e46df816ebab263926f3
                      kubernetes.io/config.seen: 2022-12-07T17:49:35.854662116+01:00
                      kubernetes.io/config.source: file
                      kubernetes.io/psp: global-unrestricted-psp
Status:               Running
IP:                   <redacted-ipv4>
IPs:
  IP:           <redacted-ipv4>
  IP:           <redacted-ipv6>
Controlled By:  Node/<redacted-host>
Containers:
  kube-controller-manager:
    Container ID:  containerd://109a2213c07d35080264d92565042dc6b1ae873d3e810eae677aa94563253ef3
    Image:         index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110
    Image ID:      docker.io/rancher/hardened-kubernetes@sha256:cd0a738ec0f435fcf0f3419e71925a08651c75096b5eb74e6930b2b92b1ad731
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-controller-manager
    Args:
      --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins
      --terminated-pod-gc-threshold=1000
      --permit-port-sharing=true
      --allocate-node-cidrs=true
      --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --bind-address=127.0.0.1
      --cluster-cidr=10.42.0.0/16
      --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt
      --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key
      --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt
      --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key
      --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt
      --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key
      --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt
      --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key
      --configure-cloud-routes=false
      --controllers=*,-service,-route,-cloud-node-lifecycle
      --feature-gates=JobTrackingWithFinalizers=true
      --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --profiling=false
      --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt
      --secure-port=10257
      --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key
      --service-cluster-ip-range=10.43.0.0/16
      --use-service-account-credentials=true
    State:          Running
      Started:      Wed, 07 Dec 2022 22:08:44 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 07 Dec 2022 22:08:21 +0100
      Finished:     Wed, 07 Dec 2022 22:08:22 +0100
    Ready:          True
    Restart Count:  41
    Requests:
      cpu:     200m
      memory:  256Mi
    Liveness:  http-get https://localhost:10257/healthz delay=10s timeout=15s period=10s #success=1 #failure=8
    Startup:   http-get https://localhost:10257/healthz delay=10s timeout=5s period=10s #success=1 #failure=24
    Environment:
      FILE_HASH:  b54bf39627ac4c055b22cf098919cf51985f90ece31177432ae83b684660ae7f
      NO_PROXY:   .svc,.cluster.local,10.42.0.0/16,10.43.0.0/16
      POD_HASH:   d9722f2e48ca98d57d5ad9ca4c7e1cfb
    Mounts:
      /etc/pki/tls/certs from dir1 (rw)
      /etc/ssl/certs from dir0 (rw)
      /var/lib/rancher/rke2/server/cred/controller.kubeconfig from file1 (ro)
      /var/lib/rancher/rke2/server/db/etcd/name from file0 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.crt from file2 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.key from file3 (ro)
      /var/lib/rancher/rke2/server/tls/client-controller.crt from file4 (ro)
      /var/lib/rancher/rke2/server/tls/client-controller.key from file5 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.crt from file6 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.key from file7 (ro)
      /var/lib/rancher/rke2/server/tls/service.key from file8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  dir0:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  dir1:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/pki/tls/certs
    HostPathType:  DirectoryOrCreate
  file0:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/db/etcd/name
    HostPathType:  File
  file1:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/cred/controller.kubeconfig
    HostPathType:  File
  file2:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/client-ca.crt
    HostPathType:  File
  file3:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/client-ca.key
    HostPathType:  File
  file4:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/client-controller.crt
    HostPathType:  File
  file5:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/client-controller.key
    HostPathType:  File
  file6:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/server-ca.crt
    HostPathType:  File
  file7:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/server-ca.key
    HostPathType:  File
  file8:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/service.key
    HostPathType:  File
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute op=Exists
Events:
  Type     Reason          Age                    From     Message
  ----     ------          ----                   ----     -------
  Normal   SandboxChanged  3h21m                  kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         3h21m (x2 over 3h21m)  kubelet  Back-off restarting failed container
  Normal   Pulled          3h20m (x2 over 3h21m)  kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Created         3h20m (x2 over 3h21m)  kubelet  Created container kube-controller-manager
  Normal   Started         3h20m (x2 over 3h21m)  kubelet  Started container kube-controller-manager
  Normal   SandboxChanged  3h18m                  kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         3h18m (x3 over 3h18m)  kubelet  Back-off restarting failed container
  Normal   Pulled          3h17m (x2 over 3h18m)  kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Created         3h17m (x2 over 3h18m)  kubelet  Created container kube-controller-manager
  Normal   Started         3h17m (x2 over 3h18m)  kubelet  Started container kube-controller-manager
  Normal   SandboxChanged  3h15m                  kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         3h15m (x3 over 3h15m)  kubelet  Back-off restarting failed container
  Normal   Pulled          3h14m (x2 over 3h15m)  kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Created         3h14m (x2 over 3h15m)  kubelet  Created container kube-controller-manager
  Normal   Started         3h14m (x2 over 3h15m)  kubelet  Started container kube-controller-manager
  Normal   SandboxChanged  3h12m                  kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         3h11m (x3 over 3h12m)  kubelet  Back-off restarting failed container
  Normal   Pulled          3h11m (x2 over 3h12m)  kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Created         3h11m (x2 over 3h12m)  kubelet  Created container kube-controller-manager
  Normal   Started         3h11m (x2 over 3h12m)  kubelet  Started container kube-controller-manager
  Normal   SandboxChanged  144m                   kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         143m (x4 over 143m)    kubelet  Back-off restarting failed container
  Normal   Created         143m (x2 over 143m)    kubelet  Created container kube-controller-manager
  Normal   Pulled          143m (x2 over 143m)    kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Started         143m (x2 over 143m)    kubelet  Started container kube-controller-manager
  Normal   SandboxChanged  136m                   kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         136m (x6 over 136m)    kubelet  Back-off restarting failed container
  Normal   Pulled          136m (x2 over 136m)    kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Created         136m (x2 over 136m)    kubelet  Created container kube-controller-manager
  Normal   Started         136m (x2 over 136m)    kubelet  Started container kube-controller-manager
  Normal   SandboxChanged  96m                    kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         95m (x2 over 95m)      kubelet  Back-off restarting failed container
  Normal   Created         95m (x2 over 96m)      kubelet  Created container kube-controller-manager
  Normal   Started         95m (x2 over 96m)      kubelet  Started container kube-controller-manager
  Normal   Pulled          95m (x2 over 96m)      kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   SandboxChanged  84m                    kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         83m (x8 over 84m)      kubelet  Back-off restarting failed container
  Normal   Pulled          83m (x3 over 84m)      kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Created         83m (x3 over 84m)      kubelet  Created container kube-controller-manager
  Normal   Started         83m (x3 over 84m)      kubelet  Started container kube-controller-manager
  Normal   Created         68m                    kubelet  Created container kube-controller-manager
  Normal   Started         68m                    kubelet  Started container kube-controller-manager
  Normal   Pulled          68m                    kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Killing         63m                    kubelet  Container kube-controller-manager definition changed, will be restarted
  Warning  BackOff         62m (x10 over 63m)     kubelet  Back-off restarting failed container
  Normal   Pulled          62m (x3 over 63m)      kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Created         62m (x3 over 63m)      kubelet  Created container kube-controller-manager
  Normal   Started         62m (x3 over 63m)      kubelet  Started container kube-controller-manager
  Normal   Killing         43m                    kubelet  Container kube-controller-manager definition changed, will be restarted
  Warning  BackOff         42m (x9 over 43m)      kubelet  Back-off restarting failed container
  Normal   Pulled          42m (x3 over 43m)      kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Started         42m (x3 over 43m)      kubelet  Started container kube-controller-manager
  Normal   Created         42m (x3 over 43m)      kubelet  Created container kube-controller-manager
  Warning  BackOff         27m (x5 over 27m)      kubelet  Back-off restarting failed container
  Normal   Pulled          27m (x2 over 27m)      kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.24.8-rke2r1-build20221110" already present on machine
  Normal   Created         27m (x2 over 27m)      kubelet  Created container kube-controller-manager
  Normal   Started         27m (x2 over 27m)      kubelet  Started container kube-controller-manager
  Normal   Killing         9m41s                  kubelet  Container kube-controller-manager definition changed, will be restarted
  Warning  BackOff         9m6s (x9 over 9m38s)   kubelet  Back-off restarting failed container
  Normal   Created         8m51s (x3 over 9m40s)  kubelet  Created container kube-controller-manager
  Normal   Started         8m51s (x3 over 9m40s)  kubelet  Started container kube-controller-manager
  Normal   Pulled          8m51s (x3 over 9m40s)  kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.25.4-rke2r1-build20221110" already present on machine
  Normal   Killing         6m37s                  kubelet  Container kube-controller-manager definition changed, will be restarted
  Warning  BackOff         6m16s (x8 over 6m35s)  kubelet  Back-off restarting failed container
  Normal   Pulled          6m3s (x3 over 6m37s)   kubelet  Container image "index.docker.io/rancher/hardened-kubernetes:v1.25.4-rke2r1-build20221110" already present on machine
  Normal   Created         6m3s (x3 over 6m37s)   kubelet  Created container kube-controller-manager
  Normal   Started         6m3s (x3 over 6m37s)   kubelet  Started container kube-controller-manager

0 replies

brandond · 2022-12-08T05:21:06Z

brandond
Dec 8, 2022
Maintainer

You don't appear to have any ipv6 cidrs allocated for your nodes:

spec:
    podCIDR: 10.42.0.0/24
    podCIDRs:
    - 10.42.0.0/24
    providerID: rke2://<redacted-host>

I noticed you said that you Restarted RKE2 systemctl restart rke2-server.service. Did you add the ipv6 configuration before starting rke2 for the first time, or after? You cannot make this change after the cluster has been started.

0 replies

brandond · 2022-12-08T05:24:05Z

brandond
Dec 8, 2022
Maintainer

I also see that the controller-manager args are showing only a single ipv4 cidr: --cluster-cidr=10.42.0.0/16. This affirms my suspicion that you did not make your configuration changes before starting rke2.

0 replies

proea · 2022-12-08T07:03:15Z

proea
Dec 8, 2022
Author

I did changes after the first start. I checked the documentation and did not find that it can not be changed after the first start.

0 replies

brandond · 2022-12-08T07:48:19Z

brandond
Dec 8, 2022
Maintainer

CNI selection and cluster/service CIDRs cannot be changed once the cluster has already been started. We can leave this open to track adding this to the docs if you'd like.

0 replies

fashberg · 2023-10-07T15:36:39Z

fashberg
Oct 7, 2023

Hey!
is there any possibility to change from SingleStack to DualStack? Doing manually.

I've upgraded successfully from k3s to rke2 because the rke2 docs are not stating that it can't be changed after initial setup and now i've stuck in the same situation.
:(

Folke

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dual-stack not working #3757

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Dual-stack not working #3757

proea Dec 7, 2022

Replies: 6 comments

proea Dec 7, 2022 Author

brandond Dec 8, 2022 Maintainer

brandond Dec 8, 2022 Maintainer

proea Dec 8, 2022 Author

brandond Dec 8, 2022 Maintainer

fashberg Oct 7, 2023

proea
Dec 7, 2022

proea
Dec 7, 2022
Author

brandond
Dec 8, 2022
Maintainer

brandond
Dec 8, 2022
Maintainer

proea
Dec 8, 2022
Author

brandond
Dec 8, 2022
Maintainer

fashberg
Oct 7, 2023