Unable to pull rancher/mirrored-pause as PodSandBox from a secured registry

[tobiasf88]

Environmental Info:
K3s Version:
1.22.2+k3s2

Node(s) CPU architecture, OS, and Version:
Linux k3d-devcluster-server-0 3.10.0-1160.45.1.el7.x86_64 #1 SMP Fri Sep 24 10:17:16 UTC 2021 x86_64 GNU/Linux

Cluster Configuration:
1 Agent, 1 Server setup with k3d

Describe the bug:
I have a secured mirror registry reachable from the cluster. The Server has only access to the company network and no access the the public internet. When I start the cluster and connect to it, no pods can be initialized because it failed to pull the rancher/mirrored-pause:3.1 image, while setting up the pod:

FailedCreatePodSandBox:  failed to pull and unpack
 image "docker.io/rancher/mirrored-pause:3.1":
 failed to resolve reference 
"docker.io/rancher/mirrored-pause:3.1": failed to authorize:
 failed to fetch anonymous token: unexpected status: 401 Unauthorized 

I believe to correct setting up the cluster with k3d. I will print you the configs:

How the registires.yaml ist configured: /etc/rancher/k3s/registries.yaml -

mirrors:
  docker.io:
    endpoint:
    - https://docker-public.docker.XYZ.com
configs:
  docker.io:
    auth:
      username: USERNAME
      password: PASSWORD
      auth: BASICAUTH
      identity_token: ""
    tls:
      ca_file: ""
      cert_file: ""
      key_file: ""
      insecure_skip_verify: true

I've also checked, if i can pull the images with containred from the server. I works fine:

ctr --debug i pull -u USERNAME:PASSWORD docker-public.docker.XYZ.com/rancher/mirrored-pause:3.1

I've checkd the contaiered config for k3s /var/lib/rancher/k3s/agent/etc/containerd/config.toml

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "rancher/mirrored-pause:3.1"

[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.cri.cni]
  bin_dir = "/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.cri.registry.mirrors]

[plugins.cri.registry.mirrors."docker.io"]
  endpoint = ["https://docker-public.docker.XYZ.com/"]

[plugins.cri.registry.configs."docker.io".auth]
  username = "USERNAME"
  password = "PASSWORD"
  auth = "BASICAUTH"

[plugins.cri.registry.configs."docker.io".tls]
  insecure_skip_verify = true

Expected behavior:
authenicate to the registry and sucessfully pull the rancher/mirrored-pause:3.1 from the mirrored registry

Actual behavior:
FailedCreatePodSandBox - 401 Unauthorized

Additional context / logs:
FailedCreatePodSandBox: failed to pull and unpack
image "docker.io/rancher/mirrored-pause:3.1":
failed to resolve reference
"docker.io/rancher/mirrored-pause:3.1": failed to authorize:
failed to fetch anonymous token: unexpected status: 401 Unauthorized

Backporting

• Needs backporting to older releases

[brandond]

• mirrors determines endpoints for a registry
• configs determines credentials and tls for an endpoint

You've asked to use docker-public.docker.XYZ.com as your local registry mirror, but haven't configured any credentials for it; you've only configured credentials for docker.io. If you want to provide credentials for your internal registry, you should specify it in the configs section:

mirrors:
  docker.io:
    endpoint:
    - https://docker-public.docker.XYZ.com
configs:
  docker-public.docker.XYZ.com:
    auth:
      username: USERNAME
      password: PASSWORD
      auth: BASICAUTH
      identity_token: ""
    tls:
      ca_file: ""
      cert_file: ""
      key_file: ""
      insecure_skip_verify: true

[tobiasf88]

Sorry for late response. You are right, it worked now. It was a bit confusing.
Thank you!!!