k3s not honoring /etc/rancher/k3s/registries.yaml

[matthiasLetsGit]

Environmental Info:
K3s Version:
k3s -v
k3s version v1.28.7+k3s1 (051b14b)
go version go1.21.7
(same behavior with k3s_v1.26.14+k3s1 and k3s_v1.29.3+k3s1)

Node(s) CPU architecture, OS, and Version:
Linux k3s-node2.local 5.14.0-362.18.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 29 07:05:48 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server, 2 agents

Describe the bug:
I created my own, modified container image for nginx and pushed it to my own registry (insecure, DNS = container-registry.internal / IP = 192.168.225.11, Port 5000).
Via plain docker I can pull and execute this image.
I setup /etc/rancher/k3s/registries.yaml so that a "dummy registry" is mapped to my own registry, so that uses the entry in registries.yaml.
When I try to run a deployment via k3s in which I use my own nginx-image (mmnginxhostname) registries.yaml seems not to be honored.

Steps To Reproduce:

• Installed K3s: without flags, just: curl -sfL https://get.k3s.io | sh -s - server --cluster-init
• cat /etc/rancher/k3s/registries.yaml

configs: {}
mirrors:
  container-registry.mm:
    endpoint:
    - http://container-registry.internal:5000
    - http://192.168.225.11:5000

• kubectl get deployments mm-hostname -o yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "10"
  creationTimestamp: "2024-04-03T16:13:47Z"
  generation: 47
  labels:
    workload.user.cattle.io/workloadselector: apps.deployment-default-mm-hostname
  name: mm-hostname
  namespace: default
  resourceVersion: "109081"
  uid: 4a126bcb-e79b-48d4-a861-2a618a8d4cf1
spec:
  progressDeadlineSeconds: 600
  replicas: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: apps.deployment-default-mm-hostname
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        workload.user.cattle.io/workloadselector: apps.deployment-default-mm-hostname
      namespace: default
    spec:
      containers:
      - image: container-registry.mm/mmnginxhostname:1.25.4
        imagePullPolicy: Always
        name: container-0
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          readOnlyRootFilesystem: false
          runAsNonRoot: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

when I scale this to 1, I get "no such host" (this):

cat /var/lib/rancher/k3s/agent/containerd/containerd.log

time="2024-04-04T20:37:48.417017901+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:mm-hostname-65d6cd86cd-fc2vc,Uid:2ff805ec-a494-46a9-acf6-be70c2ea0d92,Namespace:default,Attempt:0,}"

map[string]interface {}{"cniVersion":"1.0.0", "forceAddress":true, "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[string]interface {}{map[string]interface {}{"subnet":"10.42.1.0/24"}}}, "routes":[]types.Route{types.Route{Dst:net.IPNet{IP:net.IP{0xa, 0x2a, 0x0, 0x0}, Mask:net.IPMask{0xff, 0xff, 0x0, 0x0}}, GW:net.IP(nil)}}, "type":"host-local"}, "isDefaultGateway":true, "isGateway":true, "mtu":(*uint)(0xc000012860), "name":"cbr0", "type":"bridge"}
delegateAdd: netconf sent to delegate plugin:
{"cniVersion":"1.0.0","forceAddress":true,"hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.42.1.0/24"}]],"routes":[{"dst":"10.42.0.0/16"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":1450,"name":"cbr0","type":"bridge"}time="2024-04-04T20:37:48.445894670+02:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2024-04-04T20:37:48.446182268+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2024-04-04T20:37:48.446190668+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-04-04T20:37:48.446241567+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-04-04T20:37:48.475734331+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:mm-hostname-65d6cd86cd-fc2vc,Uid:2ff805ec-a494-46a9-acf6-be70c2ea0d92,Namespace:default,Attempt:0,} returns sandbox id \"97bc91ea5bfd8f82d40d572c2ec5f4c8bbbc92e15023a6ff59caa24e6704a0ea\""
time="2024-04-04T20:37:48.476949621+02:00" level=info msg="PullImage \"container-registry.mm/mmnginxhostname:1.25.4\""
time="2024-04-04T20:37:48.741126804+02:00" level=info msg="trying next host" error="failed to do request: Head \"https://container-registry.mm/v2/mmnginxhostname/manifests/1.25.4\": dial tcp: lookup container-registry.mm: no such host" host=container-registry.mm
time="2024-04-04T20:37:48.747815350+02:00" level=error msg="PullImage \"container-registry.mm/mmnginxhostname:1.25.4\" failed" error="failed to pull and unpack image \"container-registry.mm/mmnginxhostname:1.25.4\": failed to resolve reference \"container-registry.mm/mmnginxhostname:1.25.4\": failed to do request: Head \"https://container-registry.mm/v2/mmnginxhostname/manifests/1.25.4\": dial tcp: lookup container-registry.mm: no such host"

Expected behavior:
I expect k3s to use the container-registry defined in /etc/rancher/k3s/registries.yaml

Actual behavior:
k3s is not honoring registries.yaml

Additional context / logs:
Log already at "Steps To Reproduce" provided

[brandond]

Can you confirm that you configured registries.yaml on all nodes - both servers and agents? You need to configure the containerd registry mirrors on all nodes, not just the server. The containerd log suggests the node the pod is running on does not have any mirrors configured.

You can also check /var/lib/rancher/k3s/agent/etc/containerd/certs.d/container-registry.mm/hosts.toml on the node the pod is running on. If this file does not exist, it indicates there was a problem with your registries.yaml and the mirrors have not been properly configured.

You might also note the behavior described here: https://docs.k3s.io/installation/private-registry#default-endpoint-fallback - if your endpoints are not reachable, containerd will by default fall back to trying the registry by hostname.

[matthiasLetsGit]

I can confirm this. I am aware of the requirement, that registries.yaml has to be on all nodes.

I went over everything once more.

Now I have this:

cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/container-registry.mm/hosts.toml
# File generated by k3s. DO NOT EDIT.
server = "https://container-registry.mm/v2"

[host."http://container-registry.internal:5000/v2"]
  capabilities = ["pull", "resolve"]
[host."http://192.168.225.11:5000/v2"]
  capabilities = ["pull", "resolve"]

cat /var/lib/rancher/k3s/agent/containerd/containerd.log
time="2024-04-04T22:35:21.851976480+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:mm-hostname-65d6cd86cd-s9pcf,Uid:6a30fcae-e33c-447e-a9fe-13a8a4a3a9d3,Namespace:default,Attempt:0,}"

map[string]interface {}{"cniVersion":"1.0.0", "forceAddress":true, "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[string]interface {}{map[string]interface {}{"subnet":"10.42.1.0/24"}}}, "routes":[]types.Route{types.Route{Dst:net.IPNet{IP:net.IP{0xa, 0x2a, 0x0, 0x0}, Mask:net.IPMask{0xff, 0xff, 0x0, 0x0}}, GW:net.IP(nil)}}, "type":"host-local"}, "isDefaultGateway":true, "isGateway":true, "mtu":(*uint)(0xc0000b0810), "name":"cbr0", "type":"bridge"}
delegateAdd: netconf sent to delegate plugin:
{"cniVersion":"1.0.0","forceAddress":true,"hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.42.1.0/24"}]],"routes":[{"dst":"10.42.0.0/16"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":1450,"name":"cbr0","type":"bridge"}time="2024-04-04T22:35:21.875631972+02:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2024-04-04T22:35:21.875876570+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2024-04-04T22:35:21.875896570+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-04-04T22:35:21.875934069+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-04-04T22:35:21.901376445+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:mm-hostname-65d6cd86cd-s9pcf,Uid:6a30fcae-e33c-447e-a9fe-13a8a4a3a9d3,Namespace:default,Attempt:0,} returns sandbox id \"8f90b09cca1e80ddf52fbcf041ae7f78b4431e65d5748863f9a4907d2abafdc2\""
time="2024-04-04T22:35:21.902130539+02:00" level=info msg="PullImage \"container-registry.mm/mmnginxhostname:1.25.4\""
time="2024-04-04T22:35:21.906037504+02:00" level=info msg="trying next host - response was http.StatusNotFound" host="container-registry.internal:5000"
time="2024-04-04T22:35:21.907313993+02:00" level=info msg="trying next host - response was http.StatusNotFound" host="192.168.225.11:5000"
time="2024-04-04T22:35:21.907694190+02:00" level=info msg="trying next host" error="failed to do request: Head \"https://container-registry.mm/v2/mmnginxhostname/manifests/1.25.4\": dial tcp: lookup container-registry.mm: no such host" host=container-registry.mm
time="2024-04-04T22:35:21.908788280+02:00" level=error msg="PullImage \"container-registry.mm/mmnginxhostname:1.25.4\" failed" error="failed to pull and unpack image \"container-registry.mm/mmnginxhostname:1.25.4\": failed to resolve reference \"container-registry.mm/mmnginxhostname:1.25.4\": failed to do request: Head \"https://container-registry.mm/v2/mmnginxhostname/manifests/1.25.4\": dial tcp: lookup container-registry.mm: no such host"
time="2024-04-04T22:35:21.908831480+02:00" level=info msg="stop pulling image container-registry.mm/mmnginxhostname:1.25.4: active requests=0, bytes read=0"

[brandond]

time="2024-04-04T22:35:21.906037504+02:00" level=info msg="trying next host - response was http.StatusNotFound" host="container-registry.internal:5000"
time="2024-04-04T22:35:21.907313993+02:00" level=info msg="trying next host - response was http.StatusNotFound" host="192.168.225.11:5000"
time="2024-04-04T22:35:21.907694190+02:00" level=info msg="trying next host" error="failed to do request: Head \"https://container-registry.mm/v2/mmnginxhostname/manifests/1.25.4\": dial tcp: lookup container-registry.mm: no such host" host=container-registry.mm

Neither of your mirrors had the image, so it tried the default endpoint (which doesn't exist), and that failed too. Sounds like it's working as expected.

[matthiasLetsGit]

OK. Sorry for bothering you.

But I was not able to find the appropriate logs.

You, asking the right questions, pointed me to the right spots.

Big Thanks!