Attempting to use device_ownership_from_security_context = true

[Smithx10]

I am attempting to use the setting: "device_ownership_from_security_context = true" in order to allow KubeVirt CDI to import Block devices instead of filesystem PVs. https://github.com/kubevirt/containerized-data-importer/blob/main/doc/block_cri_ownership_config.md

When I configure this on a new RKE2-Agent containerd pods go into the "EXIT" state.

Exit State

root@rke2-a1:~# crictl ps  -a
I0303 05:07:55.575339   46643 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/run/k3s/containerd/containerd.sock" URL="unix:///run/k3s/containerd/containerd.sock"
I0303 05:07:55.578800   46643 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/run/k3s/containerd/containerd.sock" URL="unix:///run/k3s/containerd/containerd.sock"
CONTAINER           IMAGE               CREATED              STATE               NAME                               ATTEMPT             POD ID              POD
071c3f9842d37       d7cbff5e50189       24 seconds ago       Exited              install-multus-binary              6                   7566dcfe7958b       kube-multus-ds-f5696
ef4240a0757b7       f84707403d427       About a minute ago   Exited              csi-node-driver-registrar          5                   7a8e41338563d       openebs-zfs-node-cdlgg
d6cb424f65ade       d7cbff5e50189       2 minutes ago        Exited              kube-multus                        5                   3cdf37ba2aaae       kube-multus-ds-f5696
4f2298a3506f0       b6c5c3537ae76       4 minutes ago        Exited              install-cni                        4                   ab298b91505b8       kube-ovn-cni-6zfcx
91970cf5c4bfd       b6c5c3537ae76       4 minutes ago        Exited              cni-server                         6                   93ddac90d4d80       kube-ovn-cni-6zfcx
88334070833b7       b6c5c3537ae76       4 minutes ago        Exited              openvswitch                        6                   095e3e991fc9a       ovs-ovn-kbfbg
6a86ee9682f44       ab7fea56055ba       4 minutes ago        Exited              kube-proxy                         6                   e26faa90706bb       kube-proxy-rke2-a1
f027de88d9b67       26e1a60435b95       5 minutes ago        Exited              openebs-zfs-plugin                 5                   46b99ec893965       openebs-zfs-node-cdlgg
5c11afd487b36       9051c01da7c50       10 minutes ago       Exited              metrics-server                     3                   6fcfc4791e6b6       rke2-metrics-server-544c8c66fc-qzjv7
9d8d7038d5593       5f89cb8137ccb       10 minutes ago       Exited              helm                               3                   85702e3948e54       helm-install-rke2-snapshot-controller-crd-wjgbc
e1ab7d50a5ac1       c0ae43f67f3f9       10 minutes ago       Exited              virt-handler                       1                   74515c6374751       virt-handler-hk98w
d262effd076f7       1b5aedc400f2a       10 minutes ago       Exited              autoscaler                         1                   6b769a11ec9ce       rke2-coredns-rke2-coredns-autoscaler-945fbd459-pd9fk
48266bd0db7b8       ff52c2bcf9f88       10 minutes ago       Exited              rke2-snapshot-validation-webhook   1                   badd56611f74f       rke2-snapshot-validation-webhook-54c5989b65-x7dpf
98d0e3da2e6bf       b6c5c3537ae76       11 minutes ago       Exited              pinger                             1                   6d8ed190b8403       kube-ovn-pinger-zj49w
183dbd5171593       1ef6c138bd5f2       11 minutes ago       Exited              rke2-snapshot-controller           1                   c0a24866a140c       rke2-snapshot-controller-59cc9cd8f4-xk7hx

ContainerD Logs

time="2024-03-03T05:07:32.213558910Z" level=error msg="Failed to destroy network for sandbox \"e3975fbedcf5cf70771d104edd3652a2830e8754445976873e80aebc1a0e752e\"" error="plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:32.213927837Z" level=error msg="encountered an error cleaning up failed sandbox \"e3975fbedcf5cf70771d104edd3652a2830e8754445976873e80aebc1a0e752e\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:32.213974405Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:helm-install-rke2-snapshot-controller-crd-wjgbc,Uid:c9d44e46-2ca7-4212-a8e8-675ebf3ff306,Namespace:kube-system,Attempt:3,} failed, error" error="failed to setup network for sandbox \"e3975fbedcf5cf70771d104edd3652a2830e8754445976873e80aebc1a0e752e\": plugin type=\"kube-ovn\" failed (add): RPC failed; Post \"http://dummy/api/v1/add\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:32.239129654Z" level=error msg="Failed to destroy network for sandbox \"a60d20421d0030f20d7542f486aba20cb9c7117c669823a256508d0a153e662a\"" error="plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:32.239517611Z" level=error msg="encountered an error cleaning up failed sandbox \"a60d20421d0030f20d7542f486aba20cb9c7117c669823a256508d0a153e662a\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:32.239569399Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:rke2-metrics-server-544c8c66fc-qzjv7,Uid:13c9b352-d8b4-4e50-8b83-de19db9af6bf,Namespace:kube-system,Attempt:5,} failed, error" error="failed to setup network for sandbox \"a60d20421d0030f20d7542f486aba20cb9c7117c669823a256508d0a153e662a\": plugin type=\"kube-ovn\" failed (add): RPC failed; Post \"http://dummy/api/v1/add\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:32.521159828Z" level=info msg="shim disconnected" id=071c3f9842d37eaf06912ed3941f49e14c1499edb8002ec53b2db81e1e53888a namespace=k8s.io
time="2024-03-03T05:07:32.521220737Z" level=warning msg="cleaning up after shim disconnected" id=071c3f9842d37eaf06912ed3941f49e14c1499edb8002ec53b2db81e1e53888a namespace=k8s.io
time="2024-03-03T05:07:32.521228776Z" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2024-03-03T05:07:32.850154003Z" level=info msg="StopPodSandbox for \"85702e3948e54d177202e76a9ba9219fc4ddf131dcaa9ff6fd6e0ef769af4a48\""
time="2024-03-03T05:07:32.850231350Z" level=info msg="Container to stop \"9d8d7038d5593501d9cc029c6559306d821b98ed1671fd2aab5f1311b4f80f6a\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
time="2024-03-03T05:07:32.853017100Z" level=info msg="StopPodSandbox for \"6fcfc4791e6b6e9fab0dfdd1ab870de6aa84c196696cf1bf43c53a6a8a9f396b\""
time="2024-03-03T05:07:32.853098808Z" level=info msg="Container to stop \"5c11afd487b36bc26a46d011a2e51fd0ceff0310b7709e52f4f98e729e53c637\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
time="2024-03-03T05:07:32.873095171Z" level=info msg="TearDown network for sandbox \"85702e3948e54d177202e76a9ba9219fc4ddf131dcaa9ff6fd6e0ef769af4a48\" successfully"
time="2024-03-03T05:07:32.873141888Z" level=info msg="StopPodSandbox for \"85702e3948e54d177202e76a9ba9219fc4ddf131dcaa9ff6fd6e0ef769af4a48\" returns successfully"
time="2024-03-03T05:07:32.873755767Z" level=info msg="StopPodSandbox for \"e3975fbedcf5cf70771d104edd3652a2830e8754445976873e80aebc1a0e752e\""
time="2024-03-03T05:07:32.874010658Z" level=info msg="Ensure that sandbox e3975fbedcf5cf70771d104edd3652a2830e8754445976873e80aebc1a0e752e in task-service has been cleanup successfully"
time="2024-03-03T05:07:32.875624810Z" level=info msg="TearDown network for sandbox \"6fcfc4791e6b6e9fab0dfdd1ab870de6aa84c196696cf1bf43c53a6a8a9f396b\" successfully"
time="2024-03-03T05:07:32.875669868Z" level=info msg="StopPodSandbox for \"6fcfc4791e6b6e9fab0dfdd1ab870de6aa84c196696cf1bf43c53a6a8a9f396b\" returns successfully"
time="2024-03-03T05:07:32.876998560Z" level=info msg="StopPodSandbox for \"a60d20421d0030f20d7542f486aba20cb9c7117c669823a256508d0a153e662a\""
time="2024-03-03T05:07:32.877695766Z" level=info msg="Ensure that sandbox a60d20421d0030f20d7542f486aba20cb9c7117c669823a256508d0a153e662a in task-service has been cleanup successfully"
time="2024-03-03T05:07:32.897178798Z" level=error msg="StopPodSandbox for \"e3975fbedcf5cf70771d104edd3652a2830e8754445976873e80aebc1a0e752e\" failed" error="failed to destroy network for sandbox \"e3975fbedcf5cf70771d104edd3652a2830e8754445976873e80aebc1a0e752e\": plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:32.898880667Z" level=error msg="StopPodSandbox for \"a60d20421d0030f20d7542f486aba20cb9c7117c669823a256508d0a153e662a\" failed" error="failed to destroy network for sandbox \"a60d20421d0030f20d7542f486aba20cb9c7117c669823a256508d0a153e662a\": plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:34.136765481Z" level=info msg="StopPodSandbox for \"6d8ed190b8403e7d1034b7da680b3cd9a10226acaba2fa318e21eb6341cf2743\""
time="2024-03-03T05:07:34.136901876Z" level=info msg="Container to stop \"98d0e3da2e6bf67d4b4067985e0689edda2acdd6e3d8e3ca501b01355eda9ee4\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
time="2024-03-03T05:07:34.158184634Z" level=info msg="TearDown network for sandbox \"6d8ed190b8403e7d1034b7da680b3cd9a10226acaba2fa318e21eb6341cf2743\" successfully"
time="2024-03-03T05:07:34.158235501Z" level=info msg="StopPodSandbox for \"6d8ed190b8403e7d1034b7da680b3cd9a10226acaba2fa318e21eb6341cf2743\" returns successfully"
time="2024-03-03T05:07:34.159208906Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-ovn-pinger-zj49w,Uid:9ddb57e4-f440-4920-b20a-a8b4c58f7d4d,Namespace:kube-system,Attempt:3,}"
time="2024-03-03T05:07:34.220611168Z" level=error msg="Failed to destroy network for sandbox \"e78d8c2b93687046d7a0a515437c770e387f8e9313bb82c09356413685899dd9\"" error="plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:34.223465236Z" level=error msg="encountered an error cleaning up failed sandbox \"e78d8c2b93687046d7a0a515437c770e387f8e9313bb82c09356413685899dd9\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:34.223540833Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-ovn-pinger-zj49w,Uid:9ddb57e4-f440-4920-b20a-a8b4c58f7d4d,Namespace:kube-system,Attempt:3,} failed, error" error="failed to setup network for sandbox \"e78d8c2b93687046d7a0a515437c770e387f8e9313bb82c09356413685899dd9\": plugin type=\"kube-ovn\" failed (add): RPC failed; Post \"http://dummy/api/v1/add\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:34.863140923Z" level=info msg="StopPodSandbox for \"6d8ed190b8403e7d1034b7da680b3cd9a10226acaba2fa318e21eb6341cf2743\""
time="2024-03-03T05:07:34.863209931Z" level=info msg="Container to stop \"98d0e3da2e6bf67d4b4067985e0689edda2acdd6e3d8e3ca501b01355eda9ee4\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
time="2024-03-03T05:07:34.881368161Z" level=info msg="TearDown network for sandbox \"6d8ed190b8403e7d1034b7da680b3cd9a10226acaba2fa318e21eb6341cf2743\" successfully"
time="2024-03-03T05:07:34.881416869Z" level=info msg="StopPodSandbox for \"6d8ed190b8403e7d1034b7da680b3cd9a10226acaba2fa318e21eb6341cf2743\" returns successfully"
time="2024-03-03T05:07:34.881888812Z" level=info msg="StopPodSandbox for \"e78d8c2b93687046d7a0a515437c770e387f8e9313bb82c09356413685899dd9\""
time="2024-03-03T05:07:34.882168612Z" level=info msg="Ensure that sandbox e78d8c2b93687046d7a0a515437c770e387f8e9313bb82c09356413685899dd9 in task-service has been cleanup successfully"
time="2024-03-03T05:07:34.900570243Z" level=error msg="StopPodSandbox for \"e78d8c2b93687046d7a0a515437c770e387f8e9313bb82c09356413685899dd9\" failed" error="failed to destroy network for sandbox \"e78d8c2b93687046d7a0a515437c770e387f8e9313bb82c09356413685899dd9\": plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:37.137130422Z" level=info msg="StopPodSandbox for \"6b769a11ec9cec5ca2caddc6993d15cd752b2e1f903a086ca54f522b56a287eb\""
time="2024-03-03T05:07:37.137223929Z" level=info msg="Container to stop \"d262effd076f74e489d54b74c67a89b7019d85ec3b16b99392e79cd305e85b7a\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
time="2024-03-03T05:07:37.155981886Z" level=info msg="TearDown network for sandbox \"6b769a11ec9cec5ca2caddc6993d15cd752b2e1f903a086ca54f522b56a287eb\" successfully"
time="2024-03-03T05:07:37.156030915Z" level=info msg="StopPodSandbox for \"6b769a11ec9cec5ca2caddc6993d15cd752b2e1f903a086ca54f522b56a287eb\" returns successfully"
time="2024-03-03T05:07:37.156760958Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:rke2-coredns-rke2-coredns-autoscaler-945fbd459-pd9fk,Uid:76c1ea4d-6f91-4b65-bb85-ad054abc7fa8,Namespace:kube-system,Attempt:3,}"
time="2024-03-03T05:07:37.220520025Z" level=error msg="Failed to destroy network for sandbox \"b989bb939bb7fe21048cc1312bf42ebdaca8982f2cad362e05e96a01de9797a9\"" error="plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"
time="2024-03-03T05:07:37.220931070Z" level=error msg="encountered an error cleaning up failed sandbox \"b989bb939bb7fe21048cc1312bf42ebdaca8982f2cad362e05e96a01de9797a9\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"kube-ovn\" failed (delete): RPC failed; Post \"http://dummy/api/v1/del\": dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"

ContainerD Template

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/rke2/agent/containerd"

[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "index.docker.io/rancher/mirrored-pause:3.6"
  device_ownership_from_security_context = true

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "zfs"
  disable_snapshot_annotations = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

rke2 version

root@rke2-s1:~# rke2 -v
rke2 version v1.29.0+rke2r1 (4fd30c26c91dd3f2f623c5af00d1ebcfec8c2709)
go version go1.21.5 X:boringcrypto

[brandond]

dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: no such file or directory"

The errors in your containerd log all appear to be due openvswitch stuff failing. You didn't mention deploying a custom CNI; in what other ways have you customized this cluster?

I will also note that your containerd config template contains no templating; this will likely cause additional problems in the future. You should use the actual template from the appropriate release, and change the bits you need to modify. Do not simply copy the existing rendered config into the template file.

https://github.com/k3s-io/k3s/blob/v1.29.0%2Bk3s1/pkg/agent/templates/templates_linux.go

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.