k3s rootless - two step install

[vsoch]

Hi! I'm trying to prepare a set of VMs with k3s, and have everything that I need to (later) allow a user to login and setup rootless. With the current suggested workflow, namely calling the install script in one of two ways:

# For the control plane
curl -sfL https://get.k3s.io | K3S_TOKEN=${secret_token} sh -

# For an agent
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=testing K3S_URL=https://${login_node}:6443 K3S_TOKEN=${secret_token} sh -

This assumes that I can get sudo, so I can't have this be done by a user. So I've been trying to piece apart the logic of that script to separate the two needs (root vs user run) and I found both of these:

INSTALL_K3S_SKIP_ENABLE
INSTALL_K3S_SKIP_START

And that seems to successfully install the binaries, but then when I need to generate the kubeconfig for the user, e.g.,

cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

that file isn't there. I could have the full startup script with some extra envars in the boot script, but we don't actually have hostnames yet and it's run for both control plane / agent nodes, so I can't be specific to choose which one to run. So I want to ask (and hopefully this is an easy answer!) how do I install this in a way that I can achieve the above - having the binaries / configs mostly ready to go on a system (prepared by root) and then a user just needs to setup their rootless service? Thank you!

[brandond]

Are you really talking about using k3s rootless, or are you just talking about installing and running k3s normally (as root, via a normal systemd service) and allowing users to access it without any additional privilege?

I will note that if you give a non-root user admin access to Kubernetes you might as well just give them root as well; they can fully escalate to root by creating privileged pods that mount the host filesystem.

[vsoch]

It sounds like your goal is to allow users to run K3s on multiple nodes, without the users having root on those nodes?

Yes that is correct. Right now I'm developing in a terraform deployed set of nodes that have Flux Framework installed (the job manager that allows us to do this).

I'm guessing you're also planning on automating all of this via a batch scheduling system of some sort? I'm not sure I can think of a good way to allow that, given the current limitations of rootless Kubernetes.

I've already figured out that part - here is how we do that with Flux, and inside of a Kubernetes operator, albeit we allow sudo. https://github.com/flux-framework/flux-operator/tree/main/examples/nested/k3s/basic. To get that working I had to build a custom image that already had k3s installed since we don't want to wait during the start of the Flux Operator MiniCluster (this is an indexed job that creates a Flux cluster so it's expected to come up quickly, mostly just puling a container): https://github.com/rse-ops/flux-hpc/tree/main/k3s

What I'm doing now is translating this to rootless, so I've moved outside of Kubernetes and am using Flux installed on VMs via terraform, and then a combination of boot scripts and what is called a flux batch (creating a grouped set of commands under the same space owned by the user) to launch the cluster. Aside from the issue of getting the ~/.kube/config in the right place without sudo and this last step of starting the agent, at least superficially it seems I'm pretty close. Here is what I have so far if you are interested. https://github.com/converged-computing/flux-terraform-gcp/tree/test/rootless-k3s/examples/k3s

Maybe there is still something to try? The main issue seems to be it wants a configuration:

INFO[0000] Adding server to load balancer k3s-agent-load-balancer: 10.10.0.6:6443 
INFO[0000] Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [10.10.0.6:6443] [default: 10.10.0.6:6443] 
WARN[0000] Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation. 
INFO[0000] Waiting to retrieve agent configuration; server is not ready: failed to retrieve configuration from server: https://127.0.0.1:6444/v1-k3s/config: 401 Unauthorized 

And I'll go back and look at that issue again but I'm afraid I don't understand the components deeply enough to truly parse it.

[vsoch]

There is a paragraph-long discussion of what problems I ran into last time I tried to get this working,

I see the comment you linked to, but the context is running this on one node, which is different than what I'm doing because I have one node per control plane and each agent, and a full networking setup. Sorry if I'm missing the paragraph you are referencing! This is my case:

There is also the larger issue of making this work across multiple hosts, which I think would have a bunch of challenges that I haven't even thought of yet.

[vsoch]

Could it be this? #5459 (comment)

]$ ping -M do -s 8972 $login_address
PING 10.10.0.6 (10.10.0.6) 8972(9000) bytes of data.
ping: local error: Message too long, mtu=1460
ping: local error: Message too long, mtu=1460
ping: local error: Message too long, mtu=1460
ping: local error: Message too long, mtu=1460

[brandond]

Rootless K3s runs fully isolated with its own pid/user/network namespaces. All instances of rootless k3s will use the same node IP for the user network namespace by default, although this can be overridden with the K3S_ROOTLESS_CIDR environment variable. Slirp4netns handles routing traffic between the actual host network namespace, and the user network namespace. It also handles mapping unprivileged ports on the host to privileged ports in the user network namespace.

The current unsolved challenges probably mostly revolve around how to get the Flannel CNI working through the port mapping; at the moment there is no way to tell either k3s or flannel to communicate with the peer using the 10000-port offset. It would take some engineering work across k3s and flannel to get all that working properly.

[vsoch]

Thank you it makes much more sense. The rootless setup is expecting that endpoint to be in this isolated namespace and it is not. I haven't contributed to flannel or the project here, but I'm fairly OK with Go and developing for Kubernetes and containers, so (if you are able) if you wanted to give me high level places to look and things to try, I could probably take a shot. It sounds like we would hit this issue regardless of the tool we tried (e.g., I was also looking at usernetes but haven't tried it yet) so aside from being interesting / and I'd learn a lot, it could be at least a start toward this vision. Over in HPCLand we are very interested in embracing Kubernetes, and while we do have access to clouds, the ultimate goal is to get it working on our current resources (of course with these constraints). Let me know your thoughts and if there is a logical entrypoint for me to help.

[brandond]

I would probably try to figure out what bit of it to tackle first. If your most important use case involves a rootless server and rootless agents running on different hosts, I would probably try to figure out how to get vxlan working with the port offset, and through slirp4netns. I have done zero investigation on that, @AkihiroSuda may have some suggestions.

This is all assuming that you can figure out how to coordinate using unique rootless CIDRs across all your nodes using some automation in your provisioning framework.

[vsoch]

If your most important use case involves a rootless server and rootless agents running on different hosts

Yes! That is absolutely the most important bit.

This is all assuming that you can figure out how to coordinate using unique rootless CIDRs across all your nodes using some automation in your provisioning framework.

We'll have to figure this out as well - I'm good just starting with step 1 and having this come after it.

@AkihiroSuda let me know if you have some pointers before I try to figure out these different pieces - even just links / comments or ideas would be helpful.

[AkihiroSuda]

I would probably try to figure out how to get vxlan working with the port offset, and through slirp4netns. I have done zero investigation on that, @AkihiroSuda may have some suggestions.

https://github.com/rootless-containers/usernetes/tree/v20230518.0#ip-addresses
https://github.com/rootless-containers/usernetes/blob/58df6ea63cc4a00425b80a088889015eedc96320/docker-compose.yml#L67

In usernetes, each of the nodes has to be explicitly configured with --cidr=10.0.XXX.0/24.
k3s should follow this too.