Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Calico pollutes the rke2-win logs #4722

Closed
manuelbuil opened this issue Sep 1, 2023 · 4 comments
Closed

Calico pollutes the rke2-win logs #4722

manuelbuil opened this issue Sep 1, 2023 · 4 comments
Assignees
@VestigeJ @manuelbuil
Milestone
v1.28.3+rke2r1

Comments

@manuelbuil
Copy link
Contributor

Environmental Info:
RKE2 Version:

Windows

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

Describe the bug:

Calico starts right after we start the different kubernetes components (e.g. kubelet). As a consequence, it takes a bit until everything is ready for Calico to start correctly. We are starting calico as par of an infinite loop and thus, we get ~20 times the same log:

9/1/2023 12:46:17 AM Calico exited: exit status 1 
9/1/2023 12:46:17 AM Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig K8S_SERVICE_CIDR=10.43.0.0/16 NODENAME=win-sf091q6r3vv CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes 
                     CALICO_K8S_NODE_REF=win-sf091q6r3vv CALICO_LOG_DIR= DNS_NAME_SERVERS=10.43.0.10 DNS_SEARCH=svc.cluster.local ETCD_ENDPOINTS= ETCD_KEY_FILE= ETCD_CERT_FILE= ETCD_CA_CERT_FILE= CNI_BIN_DIR=c:\var\lib\rancher\rke2\bin 
                     CNI_CONF_DIR=c:\var\lib\rancher\rke2\agent\etc\cni CNI_CONF_FILENAME= CNI_IPAM_TYPE=calico-ipam FELIX_LOGSEVERITYFILE=none FELIX_LOGSEVERITYSYS=none STARTUP_VALID_IP_TIMEOUT=90 IP=10.85.8.139
                     IP_AUTODETECTION_METHOD=first-found USE_POD_CIDR=false VXLAN_VNI=4096 CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name]

Steps To Reproduce:

  • Installed RKE2:
    Install rke2 windows with calico as the CNI plugin

Expected behavior:

See a couple of times the same log

Actual behavior:

Additional context / logs:
@manuelbuil manuelbuil self-assigned this Sep 1, 2023
@manuelbuil manuelbuil mentioned this issue Sep 1, 2023
Merged
@manuelbuil manuelbuil added this to the v1.28.2+rke2r1 milestone Sep 1, 2023
This was referenced Sep 25, 2023
Closed
Closed
Closed
@VestigeJ
Copy link
Contributor

VestigeJ commented Oct 9, 2023 •
edited
Loading

Not really seeing this much verbosity in the logs on reproduction - maybe it was more prevalent on k3s.

##Environment Details
VERSION=v1.28.2+rke2r1
COMMIT=

Infrastructure

  • Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME               STATUS   ROLES                       AGE     VERSION
ec2amaz-vv0kpng    Ready    <none>                      4m59s   v1.28.2
ip-1-1-2-160       Ready    control-plane,etcd,master   8m32s   v1.28.2+rke2r1
ip-1-1-3-213       Ready    <none>                      4m13s v1.28.2+rke2r1

Config.yaml:

write-kubeconfig-mode: 644
token: YOUR_TOKEN_HERE
cni: calico
profile: cis
selinux: true
node-external-ip: 1.1.2.127

Not Reproducible

$ curl https://get.rke2.io --output install-"rke2".sh
$ sudo chmod +x install-"rke2".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_RKE2_VERSION=v1.28.2+rke2r1 INSTALL_RKE2_EXEC=server ./install-rke2.sh
$ go_rke2 //sudo systemctl start rke2-server --now
$ set_kubefig //export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
$ w2 kg no,po -A //watch -n 2 kubectl get nodes,pods -A
$ sudo journalctl --no-pager | grep -i calico
$ get_report //generate this template

Results:

$ sudo journalctl --no-pager | grep -i calico

Oct 09 18:35:02 ip-172-31-28-160 rke2[16891]: time="2023-10-09T18:35:02Z" level=info msg="Extracting file charts/rke2-calico-crd.yaml to /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/charts/rke2-calico-crd.yaml"
Oct 09 18:35:02 ip-172-31-28-160 rke2[16891]: time="2023-10-09T18:35:02Z" level=info msg="Extracting file charts/rke2-calico.yaml to /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/charts/rke2-calico.yaml"
Oct 09 18:35:02 ip-172-31-28-160 rke2[16891]: time="2023-10-09T18:35:02Z" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values"
Oct 09 18:35:03 ip-172-31-28-160 rke2[16891]: time="2023-10-09T18:35:03Z" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values"
Oct 09 18:36:45 ip-172-31-28-160 rke2[16891]: time="2023-10-09T18:36:45Z" level=error msg="error syncing 'kube-system/rke2-calico': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-calico\" not found, requeuing"
Oct 09 18:36:46 ip-172-31-28-160 rke2[16891]: time="2023-10-09T18:36:46Z" level=error msg="error syncing 'kube-system/rke2-calico-crd': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-calico-crd\" not found, requeuing"
Oct 09 18:37:46 ip-172-31-28-160 (udev-worker)[20325]: calico_tmp_B: Could not generate persistent MAC: No data available
Oct 09 18:37:46 ip-172-31-28-160 (udev-worker)[20327]: calico_tmp_A: Could not generate persistent MAC: No data available
Oct 09 18:37:46 ip-172-31-28-160 (udev-worker)[20325]: calico_tmp_B: Process '/usr/sbin/hostinfo network' failed with exit code 4.
Oct 09 18:37:46 ip-172-31-28-160 (udev-worker)[20325]: calico_tmp_B: Process '/usr/sbin/hostinfo network' failed with exit code 4.

VS

$ sudo INSTALL_RKE2_COMMIT=302b2915ff5013cb3f566c455cfdefe54a0e8882 INSTALL_RKE2_EXEC=server ./install-rke2.sh
$ rke2 -v

rke2 version v1.28.2+dev.302b2915 (302b2915ff5013cb3f566c455cfdefe54a0e8882)
go version go1.20.8 X:boringcrypto

$ sudo journalctl --no-pager | grep -i calico

Oct 09 18:55:02 ip-172-31-28-160 rke2[7456]: time="2023-10-09T18:55:02Z" level=info msg="Extracting file charts/rke2-calico-crd.yaml to /var/lib/rancher/rke2/data/v1.28.2-dev.302b2915-bbe5b163e98c/charts/rke2-calico-crd.yaml"
Oct 09 18:55:02 ip-172-31-28-160 rke2[7456]: time="2023-10-09T18:55:02Z" level=info msg="Extracting file charts/rke2-calico.yaml to /var/lib/rancher/rke2/data/v1.28.2-dev.302b2915-bbe5b163e98c/charts/rke2-calico.yaml"
Oct 09 18:55:02 ip-172-31-28-160 rke2[7456]: time="2023-10-09T18:55:02Z" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values"
Oct 09 18:55:02 ip-172-31-28-160 rke2[7456]: time="2023-10-09T18:55:02Z" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values"
Oct 09 18:56:13 ip-172-31-28-160 rke2[7456]: time="2023-10-09T18:56:13Z" level=error msg="error syncing 'kube-system/rke2-calico': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-calico\" not found, requeuing"
Oct 09 18:56:14 ip-172-31-28-160 rke2[7456]: time="2023-10-09T18:56:14Z" level=error msg="error syncing 'kube-system/rke2-calico-crd': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-calico-crd\" not found, requeuing"
Oct 09 18:57:21 ip-172-31-28-160 (udev-worker)[9854]: calico_tmp_B: Could not generate persistent MAC: No data available
Oct 09 18:57:21 ip-172-31-28-160 (udev-worker)[10188]: calico_tmp_A: Could not generate persistent MAC: No data available
Oct 09 18:57:21 ip-172-31-28-160 (udev-worker)[9854]: calico_tmp_B: Process '/usr/sbin/hostinfo network' failed with exit code 4.
Oct 09 18:57:21 ip-172-31-28-160 (udev-worker)[9854]: calico_tmp_B: Process '/usr/sbin/hostinfo network' failed with exit code 4.

PS C:\Users\Administrator> get-eventlog -logname System -Source "rke2"

get-eventlog : No matches found
At line:1 char:1
+ get-eventlog -logname System -Source "rke2"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (:) [Get-EventLog], ArgumentException
    + FullyQualifiedErrorId : GetEventLogNoEntriesFound,Microsoft.PowerShell.Commands.GetEventLogCommand

PS C:\Users\Administrator> get-eventlog -logname System -Source "calico"

get-eventlog : No matches found
At line:1 char:1
+ get-eventlog -logname System -Source "calico"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (:) [Get-EventLog], ArgumentException
    + FullyQualifiedErrorId : GetEventLogNoEntriesFound,Microsoft.PowerShell.Commands.GetEventLogCommand

@VestigeJ
Copy link
Contributor

VestigeJ commented Oct 9, 2023

@manuelbuil unless you can offer comment I'm going to just close the related issues for this linked back here.

@manuelbuil
Copy link
Contributor Author

You were looking at logs in Windows agent right? You did not see a lot of logs before the PR?

@VestigeJ
Copy link
Contributor

VestigeJ commented Oct 10, 2023 •
edited
Loading

Better sample of windows agent logging noise below

BeforeFix $ get-eventlog -logname Application -source 'rke2' | select-object -Property TimeWritten,ReplacementStrings | format-table -wrap 
TimeWritten           ReplacementStrings
-----------           ------------------
10/10/2023 8:55:18 PM {Running RKE2 kube-proxy [--bind-address=1.1.2.88 --enable-dsr=true --feature-gates=WinDSR=true --network-name=Calico --source-vip=10.42.161.66 --cluster-cidr=10.42.0.0/16
                      --healthz-bind-address=127.0.0.1 --hostname-override=ec2amaz-7v0o4l9 --kubeconfig=C:\var\lib\rancher\rke2\agent\kubeproxy.kubeconfig --proxy-mode=kernelspace]}
10/10/2023 8:55:18 PM {Felix Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-7v0o4l9 CALICO_K8S_NODE_REF=ec2amaz-7v0o4l9 IP=1.1.2.88
                      USE_POD_CIDR=false FELIX_FELIXHOSTNAME=ec2amaz-7v0o4l9 FELIX_VXLANVNI=4096 FELIX_DATASTORETYPE=kubernetes]}
10/10/2023 8:55:18 PM {Deleting HNS policies before kube-proxy starts.}
10/10/2023 8:55:15 PM {Tunnel authorizer set Kubelet Port 10250}
10/10/2023 8:55:15 PM {Annotations and labels have been set successfully on node: ec2amaz-7v0o4l9}
10/10/2023 8:55:15 PM {Failed to set annotations and labels on node ec2amaz-7v0o4l9: nodes "ec2amaz-7v0o4l9" is forbidden: node "ec2amaz-7v0o4l9" is not allowed to modify taints}
10/10/2023 8:55:15 PM {Failed to set annotations and labels on node ec2amaz-7v0o4l9: nodes "ec2amaz-7v0o4l9" is forbidden: node "ec2amaz-7v0o4l9" is not allowed to modify taints}
10/10/2023 8:55:15 PM {Failed to set annotations and labels on node ec2amaz-7v0o4l9: Operation cannot be fulfilled on nodes "ec2amaz-7v0o4l9": the object has been modified; please apply your changes
                      to the latest version and try again}
10/10/2023 8:55:15 PM {Failed to set annotations and labels on node ec2amaz-7v0o4l9: Operation cannot be fulfilled on nodes "ec2amaz-7v0o4l9": the object has been modified; please apply your changes
                      to the latest version and try again}
10/10/2023 8:55:15 PM {Failed to set annotations and labels on node ec2amaz-7v0o4l9: Operation cannot be fulfilled on nodes "ec2amaz-7v0o4l9": the object has been modified; please apply your changes
                      to the latest version and try again}
10/10/2023 8:55:15 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-7v0o4l9 CALICO_K8S_NODE_REF=ec2amaz-7v0o4l9 IP=1.1.2.88
                      USE_POD_CIDR=false CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes
                      IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 8:55:15 PM {Calico exited: exit status 1}
10/10/2023 8:55:14 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-7v0o4l9 CALICO_K8S_NODE_REF=ec2amaz-7v0o4l9 IP=1.1.2.88
                      USE_POD_CIDR=false CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes
                      IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 8:55:14 PM {Calico exited: exit status 1}
10/10/2023 8:55:13 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-7v0o4l9 CALICO_K8S_NODE_REF=ec2amaz-7v0o4l9 IP=1.1.2.88
                      USE_POD_CIDR=false CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes
                      IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 8:55:13 PM {Calico exited: exit status 1}
10/10/2023 8:55:13 PM {can't find Calico_ep HNS endpoint, retrying, error=Endpoint Calico_ep not found}
10/10/2023 8:55:13 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-7v0o4l9 CALICO_K8S_NODE_REF=ec2amaz-7v0o4l9 IP=1.1.2.88
                      USE_POD_CIDR=false CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes
                      IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 8:55:13 PM {Calico exited: exit status 1}
10/10/2023 8:55:10 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-7v0o4l9 CALICO_K8S_NODE_REF=ec2amaz-7v0o4l9 IP=1.1.2.88
                      USE_POD_CIDR=false CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes
                      IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 8:55:10 PM {Calico exited: exit status 1}
10/10/2023 8:55:09 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-7v0o4l9 CALICO_K8S_NODE_REF=ec2amaz-7v0o4l9 IP=1.1.2.88
                      USE_POD_CIDR=false CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes
                      IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 8:55:09 PM {Calico exited: exit status 1}
10/10/2023 8:55:08 PM {WinDSR support is enabled}
10/10/2023 8:55:08 PM {HCN feature check, version={Major:13 Minor:3} supportedFeatures={Acl:{AclAddressLists:true AclNoHostRulePriority:true AclPortRanges:true AclRuleId:true} Api:{V1:true V2:true}
                      RemoteSubnet:true HostRoute:true DSR:true Slash32EndpointPrefixes:true AclSupportForProtocol252:true SessionAffinity:true IPv6DualStack:true SetPolicy:true VxlanPort:true
                      L4Proxy:true L4WfpProxy:true TierAcl:false}}
10/10/2023 8:55:07 PM {Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=ec2amaz-7v0o4l9
                      --kubeconfig=C:\var\lib\rancher\rke2\agent\kubeproxy.kubeconfig --proxy-mode=kernelspace}
10/10/2023 8:55:05 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-7v0o4l9 CALICO_K8S_NODE_REF=ec2amaz-7v0o4l9 IP=1.1.2.88
                      USE_POD_CIDR=false CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes
                      IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 8:55:05 PM {Running RKE2 kubelet [--cgroups-per-qos=false --enforce-node-allocatable= --file-check-frequency=5s --hairpin-mode=promiscuous-bridge --resolv-conf= --sync-frequency=30s
                      --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --client-ca-file=C:\var\lib\rancher\rke2\agent\client-ca.crt
                      --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=npipe:////./pipe/containerd-containerd --fail-swap-on=false
                      --healthz-bind-address=127.0.0.1 --hostname-override=ec2amaz-7v0o4l9 --kubeconfig=C:\var\lib\rancher\rke2\agent\kubelet.kubeconfig --node-labels=
                      --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=C:\var\lib\rancher\rke2\agent\pod-manifests --read-only-port=0
                      --resolv-conf=C:\var\lib\rancher\rke2\agent\etc\resolv.conf --serialize-image-pulls=false --tls-cert-file=C:\var\lib\rancher\rke2\agent\serving-kubelet.crt
                      --tls-private-key-file=C:\var\lib\rancher\rke2\agent\serving-kubelet.key]}
10/10/2023 8:55:05 PM {Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook
                      --client-ca-file=C:\var\lib\rancher\rke2\agent\client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local
                      --container-runtime-endpoint=npipe:////./pipe/containerd-containerd --eviction-hard=imagefs.available<5%,nodefs.available<5%
                      --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ec2amaz-7v0o4l9
                      --kubeconfig=C:\var\lib\rancher\rke2\agent\kubelet.kubeconfig --log-file=\var\lib\rancher\rke2\agent\logs\kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=
                      --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=C:\var\lib\rancher\rke2\agent\pod-manifests --read-only-port=0
                      --resolv-conf=C:\var\lib\rancher\rke2\agent\etc\resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL
                      --tls-cert-file=C:\var\lib\rancher\rke2\agent\serving-kubelet.crt --tls-private-key-file=C:\var\lib\rancher\rke2\agent\serving-kubelet.key}
10/10/2023 8:55:05 PM {Connecting to proxy, url=wss://3.3.3.209:9345/v1-rke2/connect}
10/10/2023 8:55:05 PM {Updated load balancer rke2-agent-load-balancer default server address -> 3.3.3.209:9345}
10/10/2023 8:55:05 PM {Getting list of apiserver endpoints from server}
10/10/2023 8:55:05 PM {containerd is now running}
10/10/2023 8:55:04 PM {Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing open //./pipe/containerd-containerd: The system
                      cannot find the file specified."}
10/10/2023 8:55:03 PM {Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing open //./pipe/containerd-containerd: The system
                      cannot find the file specified."}
10/10/2023 8:55:02 PM {Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing open //./pipe/containerd-containerd: The system
                      cannot find the file specified."}
10/10/2023 8:55:01 PM {Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing open //./pipe/containerd-containerd: The system
                      cannot find the file specified."}
10/10/2023 8:55:00 PM {Running containerd -c C:\var\lib\rancher\rke2\agent\etc\containerd\config.toml}
10/10/2023 8:55:00 PM {Logging containerd to C:\var\lib\rancher\rke2\agent\containerd\containerd.log}
10/10/2023 8:55:00 PM {Windows bootstrap okay. Exiting setup.}
10/10/2023 8:54:53 PM {Creating VXLAN network using the vxlanAdapter: Ethernet 3}
10/10/2023 8:54:53 PM {Generating HNS networks, please wait}
10/10/2023 8:54:53 PM {Extracting file bin\ctr.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\ctr.exe}
10/10/2023 8:54:52 PM {Extracting file bin\containerd.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\containerd.exe}
10/10/2023 8:54:52 PM {Extracting file bin\containerd-stress.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\containerd-stress.exe}
10/10/2023 8:54:51 PM {Extracting file bin\win-overlay.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\win-overlay.exe}
10/10/2023 8:54:50 PM {Extracting file bin\kubelet.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\kubelet.exe}
10/10/2023 8:54:49 PM {Extracting file bin\kubectl.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\kubectl.exe}
10/10/2023 8:54:48 PM {Extracting file bin\kube-proxy.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\kube-proxy.exe}
10/10/2023 8:54:48 PM {Extracting file bin\host-local.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\host-local.exe}
10/10/2023 8:54:48 PM {Extracting file bin\hns.psm1 to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\hns.psm1}
10/10/2023 8:54:48 PM {Extracting file bin\crictl.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\crictl.exe}
10/10/2023 8:54:47 PM {Extracting file bin\containerd-shim-runhcs-v1.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\containerd-shim-runhcs-v1.exe}
10/10/2023 8:54:46 PM {Extracting file bin\calico.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\calico.exe}
10/10/2023 8:54:45 PM {Extracting file bin\calico-node.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\calico-node.exe}
10/10/2023 8:54:43 PM {Extracting file bin\calico-ipam.exe to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\calico-ipam.exe}
10/10/2023 8:54:43 PM {Extracting file bin\confd\templates\peerings.ps1.template to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\templates\peerings.ps1.template}
10/10/2023 8:54:43 PM {Extracting file bin\confd\templates\blocks.ps1.template to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\templates\blocks.ps1.template}
10/10/2023 8:54:43 PM {Creating directory C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\templates}
10/10/2023 8:54:43 PM {Extracting file bin\confd\config-bgp.psm1 to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\config-bgp.psm1}
10/10/2023 8:54:43 PM {Extracting file bin\confd\config-bgp.ps1 to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\config-bgp.ps1}
10/10/2023 8:54:43 PM {Extracting file bin\confd\confd-service.ps1 to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\confd-service.ps1}
10/10/2023 8:54:43 PM {Extracting file bin\confd\conf.d\peerings.toml to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\conf.d\peerings.toml}
10/10/2023 8:54:43 PM {Extracting file bin\confd\conf.d\blocks.toml to C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\conf.d\blocks.toml}
10/10/2023 8:54:43 PM {Creating directory C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd\conf.d}
10/10/2023 8:54:43 PM {Creating directory C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin\confd}
10/10/2023 8:54:43 PM {Creating directory C:\var\lib\rancher\rke2\data\v1.28.2-rke2r1-windows-amd64-c283dc0cedeb\bin}
10/10/2023 8:54:43 PM {Pulling runtime image index.docker.io/rancher/rke2-runtime:v1.28.2-rke2r1-windows-amd64}
10/10/2023 8:54:43 PM {Failed to load runtime image index.docker.io/rancher/rke2-runtime:v1.28.2-rke2r1-windows-amd64 from tarball: no local image available for
                      index.docker.io/rancher/rke2-runtime:v1.28.2-rke2r1-windows-amd64: directory C:\var\lib\rancher\rke2\agent\images does not exist: image not found}
10/10/2023 8:54:43 PM {Failed to load runtime image index.docker.io/rancher/rke2-runtime:v1.28.2-rke2r1-windows-amd64 from tarball: no local image available for
                      index.docker.io/rancher/rke2-runtime:v1.28.2-rke2r1-windows-amd64: directory C:\var\lib\rancher\rke2\agent\images does not exist: image not found}
10/10/2023 8:54:43 PM {Host resolv.conf includes loopback or multicast nameservers - kubelet will use autogenerated resolv.conf with nameserver 8.8.8.8}
10/10/2023 8:54:42 PM {Running load balancer rke2-api-server-agent-load-balancer 127.0.0.1:6443 -> [3.3.3.209:6443] [default: 3.3.3.209:6443]}
10/10/2023 8:54:42 PM {Adding server to load balancer rke2-api-server-agent-load-balancer: 3.3.3.209:6443}
10/10/2023 8:54:42 PM {Running load balancer rke2-agent-load-balancer 127.0.0.1:6444 -> [3.3.3.209:9345] [default: 3.3.3.209:9345]}
10/10/2023 8:54:42 PM {Adding server to load balancer rke2-agent-load-balancer: 3.3.3.209:9345}
10/10/2023 8:54:42 PM {Starting rke2 agent v1.28.2+rke2r1 (7466261e4792e68baa2cc0c2afd3dcc929d72061)}
10/10/2023 8:54:42 PM {Stackdump - waiting signal at Global\stackdump-512}
AfterFix $ get-eventlog -logname Application -source 'rke2' | select-object -Property TimeWritten,ReplacementStrings | format-table -wrap 
TimeWritten            ReplacementStrings
-----------            ------------------
10/10/2023 10:39:50 PM {Running RKE2 kube-proxy [--bind-address=1.1.2.103 --enable-dsr=true --feature-gates=WinDSR=true --network-name=Calico --source-vip=10.42.172.130 --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1
                       --hostname-override=ec2amaz-lurnkuc --kubeconfig=C:\var\lib\rancher\rke2\agent\kubeproxy.kubeconfig --proxy-mode=kernelspace]}
10/10/2023 10:39:50 PM {Deleting HNS policies before kube-proxy starts.}
10/10/2023 10:39:48 PM {Felix Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-lurnkuc CALICO_K8S_NODE_REF=ec2amaz-lurnkuc IP=1.1.2.103 USE_POD_CIDR=false FELIX_FELIXHOSTNAME=ec2amaz-lurnkuc
                       FELIX_VXLANVNI=4096 FELIX_DATASTORETYPE=kubernetes]}
10/10/2023 10:39:45 PM {can't find Calico_ep HNS endpoint, retrying, error=Endpoint Calico_ep not found}
10/10/2023 10:39:45 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-lurnkuc CALICO_K8S_NODE_REF=ec2amaz-lurnkuc IP=1.1.2.103 USE_POD_CIDR=false
                       CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 10:39:45 PM {Calico exited: exit status 1. Retrying}
10/10/2023 10:39:43 PM {Tunnel authorizer set Kubelet Port 10250}
10/10/2023 10:39:42 PM {Annotations and labels have been set successfully on node: ec2amaz-lurnkuc}
10/10/2023 10:39:40 PM {WinDSR support is enabled}
10/10/2023 10:39:40 PM {HCN feature check, version={Major:13 Minor:3} supportedFeatures={Acl:{AclAddressLists:true AclNoHostRulePriority:true AclPortRanges:true AclRuleId:true} Api:{V1:true V2:true} RemoteSubnet:true HostRoute:true DSR:true
                       Slash32EndpointPrefixes:true AclSupportForProtocol252:true SessionAffinity:true IPv6DualStack:true SetPolicy:true VxlanPort:true L4Proxy:true L4WfpProxy:true TierAcl:false}}
10/10/2023 10:39:40 PM {Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=ec2amaz-lurnkuc --kubeconfig=C:\var\lib\rancher\rke2\agent\kubeproxy.kubeconfig --proxy-mode=kernelspace}
10/10/2023 10:39:39 PM {Calico Envs: [KUBE_NETWORK=Calico.* KUBECONFIG=c:\var\lib\rancher\rke2\agent\calico.kubeconfig NODENAME=ec2amaz-lurnkuc CALICO_K8S_NODE_REF=ec2amaz-lurnkuc IP=1.1.2.103 USE_POD_CIDR=false
                       CALICO_NODENAME_FILE=c:\var\lib\rancher\rke2\agent\calico_node_name CALICO_NETWORKING_BACKEND=vxlan CALICO_DATASTORE_TYPE=kubernetes IP_AUTODETECTION_METHOD=first-found VXLAN_VNI=4096]}
10/10/2023 10:39:39 PM {Running RKE2 kubelet [--cgroups-per-qos=false --enforce-node-allocatable= --file-check-frequency=5s --hairpin-mode=promiscuous-bridge --resolv-conf= --sync-frequency=30s --address=0.0.0.0 --anonymous-auth=false
                       --authentication-token-webhook=true --authorization-mode=Webhook --client-ca-file=C:\var\lib\rancher\rke2\agent\client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local
                       --container-runtime-endpoint=npipe:////./pipe/containerd-containerd --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ec2amaz-lurnkuc --kubeconfig=C:\var\lib\rancher\rke2\agent\kubelet.kubeconfig
                       --node-labels= --pod-infra-container-image=index.docker.io/rancher/mirrored-pause:3.6 --pod-manifest-path=C:\var\lib\rancher\rke2\agent\pod-manifests --read-only-port=0 --resolv-conf=C:\var\lib\rancher\rke2\agent\etc\resolv.conf
                       --serialize-image-pulls=false --tls-cert-file=C:\var\lib\rancher\rke2\agent\serving-kubelet.crt --tls-private-key-file=C:\var\lib\rancher\rke2\agent\serving-kubelet.key]}
10/10/2023 10:39:39 PM {Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --client-ca-file=C:\var\lib\rancher\rke2\agent\client-ca.crt --cloud-provider=external
                       --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=npipe:////./pipe/containerd-containerd --eviction-hard=imagefs.available<5%,nodefs.available<5%
                       --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ec2amaz-lurnkuc --kubeconfig=C:\var\lib\rancher\rke2\agent\kubelet.kubeconfig
                       --log-file=\var\lib\rancher\rke2\agent\logs\kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels= --pod-infra-container-image=index.docker.io/rancher/mirrored-pause:3.6
                       --pod-manifest-path=C:\var\lib\rancher\rke2\agent\pod-manifests --read-only-port=0 --resolv-conf=C:\var\lib\rancher\rke2\agent\etc\resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL
                       --tls-cert-file=C:\var\lib\rancher\rke2\agent\serving-kubelet.crt --tls-private-key-file=C:\var\lib\rancher\rke2\agent\serving-kubelet.key}
10/10/2023 10:39:39 PM {Connecting to proxy, url=wss://3.3.3.209:9345/v1-rke2/connect}
10/10/2023 10:39:39 PM {Updated load balancer rke2-agent-load-balancer default server address -> 3.3.3.209:9345}
10/10/2023 10:39:39 PM {Getting list of apiserver endpoints from server}
10/10/2023 10:39:39 PM {Imported images from C:\var\lib\rancher\rke2\agent\images\rke2-images.windows-amd64-db092b03c05320fce1ddaf4f37067318c9bb6286.tar.zst in 25.3012376s}
10/10/2023 10:39:39 PM {Imported docker.io/rancher/rke2-runtime:v1.28.2-dev.db092b03-windows-amd64}
10/10/2023 10:39:14 PM {Importing images from C:\var\lib\rancher\rke2\agent\images\rke2-images.windows-amd64-db092b03c05320fce1ddaf4f37067318c9bb6286.tar.zst}
10/10/2023 10:39:14 PM {containerd is now running}
10/10/2023 10:39:13 PM {Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing open //./pipe/containerd-containerd: The system cannot find the file specified."}
10/10/2023 10:39:12 PM {Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing open //./pipe/containerd-containerd: The system cannot find the file specified."}
10/10/2023 10:39:11 PM {Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing open //./pipe/containerd-containerd: The system cannot find the file specified."}
10/10/2023 10:39:10 PM {Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing open //./pipe/containerd-containerd: The system cannot find the file specified."}
10/10/2023 10:39:09 PM {Running containerd -c C:\var\lib\rancher\rke2\agent\etc\containerd\config.toml}
10/10/2023 10:39:09 PM {Logging containerd to C:\var\lib\rancher\rke2\agent\containerd\containerd.log}
10/10/2023 10:39:09 PM {Windows bootstrap okay. Exiting setup.}
10/10/2023 10:39:02 PM {Creating VXLAN network using the vxlanAdapter: Ethernet 3}
10/10/2023 10:39:02 PM {Generating HNS networks, please wait}
10/10/2023 10:39:02 PM {Extracting file bin\ctr.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\ctr.exe}
10/10/2023 10:39:01 PM {Extracting file bin\containerd.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\containerd.exe}
10/10/2023 10:39:01 PM {Extracting file bin\containerd-stress.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\containerd-stress.exe}
10/10/2023 10:38:54 PM {Extracting file bin\win-overlay.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\win-overlay.exe}
10/10/2023 10:38:53 PM {Extracting file bin\kubelet.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\kubelet.exe}
10/10/2023 10:38:52 PM {Extracting file bin\kubectl.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\kubectl.exe}
10/10/2023 10:38:51 PM {Extracting file bin\kube-proxy.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\kube-proxy.exe}
10/10/2023 10:38:51 PM {Extracting file bin\host-local.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\host-local.exe}
10/10/2023 10:38:51 PM {Extracting file bin\hns.psm1 to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\hns.psm1}
10/10/2023 10:38:51 PM {Extracting file bin\crictl.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\crictl.exe}
10/10/2023 10:38:50 PM {Extracting file bin\containerd-shim-runhcs-v1.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\containerd-shim-runhcs-v1.exe}
10/10/2023 10:38:50 PM {Extracting file bin\calico.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\calico.exe}
10/10/2023 10:38:49 PM {Extracting file bin\calico-node.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\calico-node.exe}
10/10/2023 10:38:48 PM {Extracting file bin\calico-ipam.exe to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\calico-ipam.exe}
10/10/2023 10:38:48 PM {Extracting file bin\confd\templates\peerings.ps1.template to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\templates\peerings.ps1.template}
10/10/2023 10:38:48 PM {Extracting file bin\confd\templates\blocks.ps1.template to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\templates\blocks.ps1.template}
10/10/2023 10:38:48 PM {Creating directory C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\templates}
10/10/2023 10:38:48 PM {Extracting file bin\confd\config-bgp.psm1 to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\config-bgp.psm1}
10/10/2023 10:38:48 PM {Extracting file bin\confd\config-bgp.ps1 to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\config-bgp.ps1}
10/10/2023 10:38:48 PM {Extracting file bin\confd\confd-service.ps1 to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\confd-service.ps1}
10/10/2023 10:38:48 PM {Extracting file bin\confd\conf.d\peerings.toml to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\conf.d\peerings.toml}
10/10/2023 10:38:48 PM {Extracting file bin\confd\conf.d\blocks.toml to C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\conf.d\blocks.toml}
10/10/2023 10:38:48 PM {Creating directory C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd\conf.d}
10/10/2023 10:38:48 PM {Creating directory C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin\confd}
10/10/2023 10:38:48 PM {Creating directory C:\var\lib\rancher\rke2\data\v1.28.2-dev.db092b03-windows-amd64-e15ef86db9ad\bin}
10/10/2023 10:38:29 PM {Checking local image archives in C:\var\lib\rancher\rke2\agent\images for index.docker.io/rancher/rke2-runtime:v1.28.2-dev.db092b03-windows-amd64}
10/10/2023 10:38:29 PM {Host resolv.conf includes loopback or multicast nameservers - kubelet will use autogenerated resolv.conf with nameserver 8.8.8.8}
10/10/2023 10:38:28 PM {Running load balancer rke2-api-server-agent-load-balancer 127.0.0.1:6443 -> [3.3.3.209:6443] [default: 3.3.3.209:6443]}
10/10/2023 10:38:28 PM {Adding server to load balancer rke2-api-server-agent-load-balancer: 3.3.3.209:6443}
10/10/2023 10:38:28 PM {Running load balancer rke2-agent-load-balancer 127.0.0.1:6444 -> [3.3.3.209:9345] [default: 3.3.3.209:9345]}
10/10/2023 10:38:28 PM {Adding server to load balancer rke2-agent-load-balancer: 3.3.3.209:9345}
10/10/2023 10:38:28 PM {Starting rke2 agent v1.28.2+dev.db092b03 (2c37467c726d5dbad94ca5131b09a51ac6574301)}
10/10/2023 10:38:28 PM {Stackdump - waiting signal at Global\stackdump-808}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VestigeJ VestigeJ

@manuelbuil manuelbuil

Labels
None yet
Projects
None yet
Milestone
v1.28.3+rke2r1
Development

No branches or pull requests
4 participants
@VestigeJ @manuelbuil @mdrahman-suse @caroline-suse-rancher