containerd.sock in Rootless K3s

[lrodriguez14]

Hello,

I have been trying to implement K3s Rootless, my configuration is just one node where I will have all my containers running in different pods, in the normal K3s I know that I can import my custom images using ctr images import and also I can list them using ctr images list, but with K3s Rootless I can't use these commands, I am getting an error that it can't find containerd.sock.

So, my questions are:

1. Where is the containerd.sock? It noticed that the /var/run/k3s/ location does not exists with rootless k3s.
2. How can I import my custom container images to containerd using K3s rootless?
3. Where is the images registry in K3s rootless? And how can I set a private one in K3s rootless?

Have a nice day,
-Leonardo R.

[ChristianCiach]

Great questions! I have no experience with k3s-rootless, but you got me curious.

Where is the containerd.sock?*

K3s does some linux-namespace shenanigans. I am not very experienced with namespaces and cgroups, but I was able to figure this out:

After starting k3s-rootless-service, let's take a look at the systemctl status --user k3s-rootless:

● k3s-rootless.service - k3s (Rootless)
     Loaded: loaded (/home/christianc/.config/systemd/user/k3s-rootless.service; disabled; vendor preset: disabled)
     Active: active (running) since Thu 2022-07-07 19:11:46 CEST; 2h 23min ago
   Main PID: 8902 (k3s-server)
      Tasks: 216
     Memory: 1.6G
        CPU: 17min 9.320s
     CGroup: /user.slice/user-1079.slice/[email protected]/app.slice/k3s-rootless.service

You can also see the cgroups here. Let's ignore them for now, because we are actually interested in the mnt-namespaces k3s is using.

From the output we can see that the "Main PID" is 8902. Let's see what pstree -S 8902 has to say:

k3s-server─┬─exe(cgroup,ipc,mnt,net,pid,user,uts)─┬─containerd-shim─┬─2*[fuse-overlayfs]
           │                                      │                 ├─local-path-prov(cgroup,ipc,mnt,net,pid,uts)───11*[{local-path-prov}]
           │                                      │                 ├─pause(ipc,mnt,net,pid,uts)
           │                                      │                 └─11*[{containerd-shim}]
           │                                      ├─containerd-shim─┬─2*[fuse-overlayfs]
           │                                      │                 ├─metrics-server(cgroup,ipc,mnt,net,pid,uts)───11*[{metrics-server}]
           │                                      │                 ├─pause(ipc,mnt,net,pid,uts)
           │                                      │                 └─11*[{containerd-shim}]
           │                                      ├─containerd-shim─┬─coredns(cgroup,ipc,mnt,net,pid,uts)───11*[{coredns}]
           │                                      │                 ├─2*[fuse-overlayfs]
           │                                      │                 ├─pause(ipc,mnt,net,pid,uts)
           │                                      │                 └─11*[{containerd-shim}]
           │                                      ├─containerd-shim─┬─entry(cgroup,ipc,mnt,net,pid,uts)
           │                                      │                 ├─entry(cgroup,ipc,mnt,net,pid,uts)
           │                                      │                 ├─3*[fuse-overlayfs]
           │                                      │                 ├─pause(ipc,mnt,net,pid,uts)
           │                                      │                 └─12*[{containerd-shim}]
           │                                      ├─containerd-shim─┬─2*[fuse-overlayfs]
           │                                      │                 ├─pause(ipc,mnt,net,pid,uts)
           │                                      │                 ├─traefik(cgroup,ipc,mnt,net,pid,uts)───11*[{traefik}]
           │                                      │                 └─12*[{containerd-shim}]
           │                                      ├─exe─┬─k3s─┬─containerd───17*[{containerd}]
           │                                      │     │     └─31*[{k3s}]
           │                                      │     └─10*[{exe}]
           │                                      ├─fuse-overlayfs
           │                                      └─12*[{exe}]
           ├─slirp4netns
           └─11*[{k3s-server}]

The -S option of pstree adds namespace-changes to the output. We can see that the exe process (actually /proc/self/exe) uses a bunch of different namespaces as compared to its parent process, including a new mnt namespace! We can also see that this mnt-namespace is propagated all the way down to the containerd process.

Let's call pstree again, but this time using the -p option to show the PIDs: pstree -p 8902. You will see a lot of output, but we are only interested in the first line that shows the PID of the exe process:

k3s-server(8902)─┬─exe(8941)─┬─containerd-shim(9856)─┬─fuse-overlayfs(9910)

Now we learned that the "exe"-process with the PID 8941 uses the same mnt-namespace as the containerd child process, so there is no need to search for the PID of containerd, but we could use this PID just as well.

Now let's use the PID 8941 to enter its mnt-namespace: sudo nsenter -a -t 8941 (or -m instead of -a to enter the mnt-namespace only, but not the other types). This spawns a shell inside the namespace. Let's see if we can find the containerd.sock here:

[root@energy80 /]# ls /run/k3s/containerd/
containerd.sock  containerd.sock.ttrpc  io.containerd.grpc.v1.cri  io.containerd.runtime.v1.linux  io.containerd.runtime.v2.task

There it is!

How can I import my custom container images to containerd using K3s rootless?

Actually, nsenter is not the only way to interact with the filesystem as seen by the containerd process. Using my PID 8941 from above again, we can also browse its filesystem inside /proc/8941/root/:

$ ls /proc/8941/root/run/k3s/containerd/
containerd.sock  containerd.sock.ttrpc  io.containerd.grpc.v1.cri  io.containerd.runtime.v1.linux  io.containerd.runtime.v2.task

This means that I can reference this socket directly when using ctr:

$ k3s ctr -a /proc/8941/root/run/k3s/containerd/containerd.sock containers list
CONTAINER                                                           IMAGE                                                  RUNTIME                  
09a52302fbf18eb57d355cd94531e0d5b779903fc760ee5613dfa70fb7ec9135    docker.io/rancher/mirrored-library-traefik:2.6.2       io.containerd.runc.v2    
47d06dfe32dda84a4dca903227e119636b528bdc3d1b943128e141dcfdf62589    docker.io/rancher/mirrored-metrics-server:v0.5.2       io.containerd.runc.v2    
612a26d7cae617575a49e1b0a7c3af96f908bb55ab631ed08e0570620cacf86d    docker.io/rancher/klipper-helm:v0.7.3-build20220613    io.containerd.runc.v2    
9230501bd41c251789125729a0296528880e098e05c9fb7e756b05ef36b1c2da    docker.io/rancher/klipper-helm:v0.7.3-build20220613    io.containerd.runc.v2    
a04ef8ba32c8474e2322ddf71b12b5bd3372a8f405de92d9814b87ae0afd158d    docker.io/rancher/mirrored-coredns-coredns:1.9.1       io.containerd.runc.v2    
ad186b7e129edcbc7fb01e2eec130fc4d97b5a8319485a612577dc80f0627121    docker.io/rancher/klipper-lb:v0.3.5                    io.containerd.runc.v2    
ad244c1e913f29f58fd1a9e2121ed77315326aa826dcde6bf0b2f7378f44e8f8    docker.io/rancher/local-path-provisioner:v0.0.21       io.containerd.runc.v2    
ae27eb032e6f2017c70d62a563bf9b060b6abe26b87ad7964272971ae9e14f6a    docker.io/rancher/mirrored-pause:3.6                   io.containerd.runc.v2    
af3034def9b157e8766134c06468e830e7e330447055e8939521bd14e72bc361    docker.io/rancher/mirrored-pause:3.6                   io.containerd.runc.v2    
bcf1a50a0a9ee320cb642f3241bbe7b38a410c8ea537e6d1b114dd4c4aea1d10    docker.io/rancher/mirrored-pause:3.6                   io.containerd.runc.v2    
cc16af4a7dd328026f9168f7dbf67b4f5eb0737975fcf034d63d0f39d88c48bb    docker.io/rancher/klipper-lb:v0.3.5                    io.containerd.runc.v2    
d9d16806f4d25b52911f072a959dd4307df9981b90889cd09cbc687e8e891e60    docker.io/rancher/mirrored-pause:3.6                   io.containerd.runc.v2    
dc5180be7ff93af9836e1a5f2563eaa4f0e16b0743facf726ddd298d9376853a    docker.io/rancher/mirrored-pause:3.6                   io.containerd.runc.v2    
dff6aa06979c30c75fce9044db190cda7815639181c470aeb594cfbf42bd4bee    docker.io/rancher/mirrored-pause:3.6                   io.containerd.runc.v2    
e6e0bbeec59a298c163fb8995825a6b73369a0143c4d8564a0d9a9e6a09ecafd    docker.io/rancher/mirrored-pause:3.6                   io.containerd.runc.v2  

You can probably use ctr image import to import your images, but I've never done that.

Where is the images registry in K3s rootless?*

I don't know if I understand this question. To the best of my knowledge, K3s does not include an image registry.

And how can I set a private one in K3s rootless?

Again, I may misunderstand this question. But maybe this helps anyway: You can actually deploy a private registry inside your local k3s cluster!

$ kubectl create deployment registry --image=registry:2
$ kubectl expose deployment registry --port=5000 --target-port=5000 --type LoadBalancer

You can copy images to this registry. I recommend https://github.com/containers/skopeo

$ skopeo copy  docker-daemon:mariadb:10.5.12 docker://localhost:5000/mariadb:10.5.12 --dest-tls-verify=false

Use can then deploy the pushed images:

$ kubectl create deployment mariadb --image=localhost:5000/mariadb:10.5.12

Neat!

[ChristianCiach]

Addendum about private registry configuration

If you want to know how to configure private registries when using k3s-rootless: You should be able to configure private registries by creating the file at this path: ~/.rancher/k3s/etc/rancher/k3s/registries.yaml.