/etc/rancher/rke2/registries.yaml doesn't work if $RKE2_HOME/agent/etc/containerd/config.toml.tmpl exists

[zhenlohuang]

Environmental Info:
RKE2 Version:
rke2 version v1.28.3+rke2r2 (cf6c1ae)
go version go1.20.10 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
Linux llm-installer-dev 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
single node

Describe the bug:
I found /etc/rancher/rke2/registries.yaml didn't work if $RKE2_HOME/agent/etc/containerd/config.toml.tmpl existed. $RKE2_HOME/agent/etc/containerd/config.toml was overrwritten by the config.toml.tmpl and didn't merge with registries.yaml.
/etc/rancher/rke2/registries.yaml:

mirrors:
  docker.io:
    endpoint:
      - "http://my-registry:30000"
  my-registry:
    endpoint:
      - "http://my-registry:30000"

[brandond]

config.toml.tmpl needs to actually be a template, and not just a copy of config.toml with a few changes made to it. If you don't retain the templating, the registry configuration cannot be rendered into it.

You can find the stock template here:
https://github.com/k3s-io/k3s/blob/v1.28.3+k3s2/pkg/agent/templates/templates_linux.go#L12-L135

[zhenlohuang]

Thanks for you quick response. I have one more question, for ExtraRuntimes where can I configure them ?

[brandond]

You cannot add arbitrary runtimes to that data structure. That is currently used to enable the auto-detected nvidia container runtimes, although we are working on adding support for detecting more - see k3s-io/k3s#8936