K3S registry rewrites

[crtvmn]

Hello,

I have the following configuration in my /etc/rancher/rke2/registries.yaml file:

mirrors:
  [...]
  my-registry:
    endpoint:
      - "http://localhost:30005"
      - "https://my-registry"
    rewrite:
      "^rancher/(.*)": "images-mirror/rancher/$1"

configs:
  [...]

The previous configuration returns different containerd config between v1.27.10 and v1.28.10.

For v1.27.10:

cat /var/lib/rancher/rke2/agent/etc/containerd/certs.d/my-registry/hosts.toml 
# File generated by rke2. DO NOT EDIT.


[host."http://localhost:30005/v2"]
  capabilities = ["pull", "resolve"]
  [host."http://localhost:30005/v2".rewrite]
    "^rancher/(.*)" = "images-mirror/rancher/$1"

[host."https://my-registry/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/var/lib/rancher/rke2/agent/registry-ca.crt"]
  client = [["/var/lib/rancher/rke2/agent/registry.crt", "/var/lib/rancher/rke2/agent/registry.key"]]

  [host."https://my-registry/v2".rewrite]
    "^rancher/(.*)" = "images-mirror/rancher/$1"

For v1.28.10:

cat /var/lib/rancher/rke2/agent/etc/containerd/certs.d/my-registry/hosts.toml
# File generated by rke2. DO NOT EDIT.

server = "https://my-registry/v2"
capabilities = ["pull", "resolve", "push"]


[host]
[host."http://localhost:30005/v2"]
  capabilities = ["pull", "resolve"]
  [host."http://localhost:30005/v2".rewrite]
    "^rancher/(.*)" = "images-mirror/rancher/$1"

For the v1.28.10, the generated config does not contain the rewrite for the https://my-registry endpoint, whereas it was in the v1.27.10 generated config. Why this change?

For your information, it works (rewrites for both endpoints) if I change the registries.yaml as following (duplicate last endpoint):

mirrors:
  [...]
  my-registry:
    endpoint:
      - "http://localhost:30005"
      - "https://my-registry"
      - "https://my-registry"
    rewrite:
      "^rancher/(.*)": "images-mirror/rancher/$1"
 
configs:
  [...]

or as following (inverse of the 2 endpoints):

mirrors:
  [...]
  my-registry:
    endpoint:
      - "https://my-registry"
      - "http://localhost:30005"
    rewrite:
      "^rancher/(.*)": "images-mirror/rancher/$1"
 
configs:
  [...]

Best regards,

Corentin

[brandond]

The previous behavior of rewriting pulls from the default endpoint broke the ability to fall back to pulling from the upstream registry, as containerd would rewrite the image path even when not pulling from the mirror. The intent was always to only rewrite when pulling from mirrors, NOT when pulling from the upstream itself.

mirrors:
  [...]
  my-registry:
    endpoint:
      - "http://localhost:30005"
      - "https://my-registry"
    rewrite:
      "^rancher/(.*)": "images-mirror/rancher/$1"

Note that it is not necessary to explicitly include the default endpoint (https://my-registry) in the endpoint list. containerd will always fall back to the default endpoint, if the image cannot be pulled from a mirror endpoint.

[brandond]

Also note that v1.27.10 and v1.28.10 are months apart. v1.27.10+k3s1 is from January, v1.28.10 is from May. Is there some reason you're comparing 5-month apart releases from different minors?

[crtvmn]

You're right, I don't need to list the https://my-registry in the endpoint list but I did it to have the rewrite for some images (like rancher images). If I don't add the https://my-registry in my previous configuration, the rewrite was not done and my cluster was not able to pull rancher images (system default registry is overwritten).

I do not understand why the rewrite is not applied to all listed endpoints because as you say, containerd will always fall back to the default endpoint so if a user do not add a rewrite to the default endpoint, he just have to not set the endpoint in the endpoint list. But if he wants to add the rewrite for a specific images list, he just has to add the default endpoint in the endpoint list.

Moreover, the following configurations add the rewrite on the default registry endpoint, why?

mirrors:
  [...]
  my-registry:
    endpoint:
      - "http://localhost:30005"
      - "https://my-registry"
      - "https://my-registry"
    rewrite:
      "^rancher/(.*)": "images-mirror/rancher/$1"
 
configs:
  [...]

or

mirrors:
  [...]
  my-registry:
    endpoint:
      - "https://my-registry"
      - "http://localhost:30005"
    rewrite:
      "^rancher/(.*)": "images-mirror/rancher/$1"
 
configs:
  [...]

[brandond]

If you need to use rewrites, I would probably recommend using your registry as a mirror for docker.io, instead of setting it as the system-default registry.

1. unset the system-default-registry flag in your config
2. set your registry as a mirror for docker.io:

   mirrors:
     docker.io:
       endpoint:
         - "https://my-registry"
       rewrite:
         "^rancher/(.*)": "images-mirror/rancher/$1"

Moreover, the following configurations add the rewrite on the default registry endpoint, why?

We recognize and handle an explicit entry for the default endpoint - specifically an entry whose endpoint matches the default endpoint URL, in the last position in the endpoint list. If you put the endpoint in the list twice, the second-to-last endpoint is NOT the default endpoint (as it is not last), so the rewrite is applied.

I will also note that if you list ANYTHING ELSE more than once, or list the default endpoint more than twice, it will raise an error - as the toml table format does not allow multiple entries for the same key.