How do I configure k3s server & agents to use private containerd registry for ALL docker images?

[matthewadams]

Hi all,

I'm in an air-gapped environment, followed docs for that. All three nodes are running Ubuntu 20.04 server, and they all have ufw disabled. I've been going in circles for days trying to bring the cluster up & start installing helm charts, but I've been having trouble both conceptually and practically trying to configure k3s to use a private registry. I have containerd, runc and CNI plugins installed (I followed instructions here). This means I have the default configuration for containerd, and I'm running it as a systemd service.

I've copied via scp all image tarballs onto the k3s master node and installed them with ctr. All good so far.

Here's my /etc/systemd/system/k3s.service file:

[Unit]
Description=Lightweight Kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/k3s server --debug --container-runtime-endpoint /run/containerd/containerd.sock

[Install]
WantedBy=default.target

Now, given this configuration, how do I tell the k3s master node and all agents to pull all images from containerd running on the master node? In other words, should I use the k3s server argument --system-default-registry & what should the value be? On the agent nodes, what should their configuration be to pull all images from the master node's containerd container registry?

I've been at this for days now...

[brandond]

Now, given this configuration, how do I tell the k3s master node and all agents to pull all images from containerd running on the master node?

You don't. The k3s registries.yaml support only works with the embedded containerd, and you're not using that - you're using the one that was provided by your Linux distribution.

You should uninstall containerd from your host, remove the --container-runtime-endpoint flag from k3s, and follow the instructions here: https://docs.k3s.io/installation/private-registry

On the agent nodes, what should their configuration be to pull all images from the master node's containerd container registry

That is not possible. containerd has an image store. It does not have an image registry that other nodes can pull from. If you want to have a local image registry to mirror or cache images from upstream container registries, you must set that up separate from k3s.

There is ongoing work to allow nodes to share images via a peer-to-peer mesh in #8977, but this is not currently possible with vanilla containerd or k3s.

[matthewadams]

Thanks for the response. I'm not that familiar with containerd, and I didn't realize it wasn't an image registry implementation. I was further confused because I read about k3s server option --system-default-registry and thought I could use that option for all k3s nodes whether master or agent. I now see that k3s agent doesn't offer than option. My goal is to point all nodes to a single registry.

Please confirm my understanding on the following points.

1. Master nodes (running k3s server) can obtain all images from a given registry via the --default-system-registry argument, and on those nodes, I don't need to have a registries.yaml file with a mirrors section.
2. Agent nodes (running k3s agent) must have a registries.yaml configured with a mirrors section that maps each registry (docker.io, quay.io, etc) at runtime to a different registry, with optional image rewrite rules.

If point 1 above is incorrect, then I suppose I need clarification on what --default-system-registry means; all I have to go on is (agent/runtime) Private registry to be used for all system images [$K3S_SYSTEM_DEFAULT_REGISTRY]. I guess the key word in that line is "system", about which I'm unclear.

After reading about the registries.yaml rewrite section, I noticed that each rewrite section is specific to a particular mirror. It occurred to me, given my use case of wanting to point all nodes to my registry mirror, that I'd like a higher-level, registry rewrite capability. registries.yaml could have a top-level section called rewrite that allows me to rewrite registries as opposed to image paths within registries. For example,

# proposed registries.yaml with new top-level `rewrite` section
rewrite:
  ".*": "http://registry.edge.local:5000"
mirrors:
  quay.io:
    rewrite:
      "^mongodb/(.*)": "foobar/mongodb-images/$1"

The above example would cause k3s to pull docker.io/bitnami/mongodb:7.0.3-debian-11-r4 from http://registry.edge.local:5000/bitnami/mongodb:7.0.3-debian-11-r4, but it would pull quay.io/mongodb/mongodb-community-server:7.0.3-ubuntu2204 from http://registry.edge.local:5000/foobar/mongodb-images/mongodb-community-server:7.0.3-ubuntu2204. This would also make the existing endpoint section optional if there is a top-level registry rewrite rule configured; if given, it would override the top-level registry rewrite rule.

Is this a reasonable request? Without it, I have to examine all images that I'll be using in my cluster and ensure that there's a mirrors entry for each registry (docker.io, quay.io, etc) with identical content:

mirrors:
  docker.io:
    endpoint:
      - "https://registry.edge.local:5000"
  quay.io:
    endpoint:
      - "https://registry.edge.local:5000"
  # ...

Appreciate your time and clarification.

[brandond]

--default-system-registry just sets the base registry for all of the built-in components - traefik, coredns, klipper-lb, and so on. If you want images for your workloads to pull from another registry, you would need to either explicitly specify that registry in the image field of all your pods, or configure mirrors in registries.yaml on all of your nodes.

We are not planning on adding global registry rewrites, if anything we will probably go the other direction and make rewrites endpoint-specific, as different registries may use different structures for storing mirrored images. The current configuration structure assumes that all mirrors will use the same rewrites, which is frequently not the case.

[matthewadams]

...then manually copying k3s-airgap-images-*.tar.gz in /var/lib/rancher/k3s/agent/images on master nodes means that there is no need to use --default-system-registry, correct?

[brandond]

that will get you the core k3s images preloaded into the containerd image store on startup, yes. Depending on what else you want to accomplish, additional steps may be necessary.

[matthewadams]

@brandond When are you hoping for #8977 to be generally available?

[brandond]

I am hoping to merge that for our january release cycle, but no promises until it's in.

[matthewadams]

@brandond I know you mentioned earlier in this discussion that you are not planning to support global registry rewrites, but this feature would make our lives much easier. We're in an air-gapped environment and, ahead of time, we docker pull all images across all helm charts that we'll need, then docker save them, scp the saved .tgz files over to our cluster, docker load them, then docker tag them so that <registry>/<repo>/<image>:<tag> images get renamed to 192.168.10.51:5555/<repo>/<image>:<tag>, where 192.168.10.51:5555 is the address of our private docker registry. The thing that's problematic without this feature is that we have to generate or manually edit our registries.yaml file by collecting all distinct registries referenced across all of our docker images and add a section for it in the registries.yaml's mirrors: section, and the contents of each entry are always of the same form:

# nested under mirrors:
  <registry>:
    endpoint:
      - "http://192.168.10.51:5555"

where <registry> is the name of each distinct docker registry across all of our docker images.

It would be so much easier if we could either have global registry rewrites in the registries.yaml or be given a k3s flag that is something like --global-registry-rewrite-endpoint http://192.168.10.51:5555 so that we don't have to generate our registries.yaml for each cluster based on the distinct registries across all docker images we need in the air-gapped environment.

Even with #8977 merged & released, unless I'm reading the new docs incorrectly, we still have to calculate the distinct set of docker registries and generate a registries.yaml with an empty entry for each distinct registry under the mirrors: section, as mentioned in the second paragraph at https://docs.k3s.io/installation/registry-mirror#enabling-registry-mirroring. It would be so much easier, and static, if I could tell k3s in some way that for any registry you come across, get it from registry http://192.168.10.51:5555.

I feel like we can't be the only one who'd find this feature useful, especially now with #8977 released. Looking forward to your response.

[brandond]

Our tool lets our field techs specify for a customer however many helm charts & explicit docker images they need for the air-gapped cluster, then, while our bastion host has an Internet connection, all explicit docker images as well as those required by all charts get pulled/saveed/scped/loaded/tagged into (today) a private registry or (tomorrow) k3s' embedded registry

Couldn't the tool that's doing this also generate the registries.yaml? It sounds like you do have an awareness of where the images are coming from before you disconnect from the internet.

[brandond]

Containerd does support the * (_default) mirror key as described above, and you can use that to configure a default endpoint for all regiestries. The issue is that spegel requires a list of registries to mirror images for; it does not support a wildcard as containerd does. We can take a look at adding support for that and pushing it upstream, but it is not something that is currently possible.

[matthewadams]

Couldn't the tool that's doing this also generate the registries.yaml? It sounds like you do have an awareness of where the images are coming from before you disconnect from the internet.

Yes, and it currently does, but we'd rather dispense with it in favor of a static global mirror so that as the inventory of charts & images changes, we don't have to regenerate the registries.yaml.

[matthewadams]

Containerd does support the * (_default) mirror key as described above, and you can use that to configure a default endpoint for all regiestries. The issue is that spegel requires a list of registries to mirror images for; it does not support a wildcard as containerd does. We can take a look at adding support for that and pushing it upstream, but it is not something that is currently possible.

Understood. Shall I enter an issue for this feature so that it can get prioritized? I'd even look into creating a PR if I can find the time -- it'd be my first taste of coding in Go, so it'd likely be some slow going (pun intended). 😊

[matthewadams]

@brandond I went ahead and filed a feature request for this. Thanks for considering it. 🙏🏼

[matthewadams]

Thanks so much for implementing this! @brandond, should update the answer to this discussion to reflect the new feature added in https://github.com/k3s-io/k3s/releases/tag/v1.29.3%2Bk3s1 via PR #9599 from issue #9590?