rke2 failing to pull via proxy: unrecognized challenge

[andreas-p]

I'm trying to install RKE2 v1.27.6+rke2r1 on a single Debian12 amd64 node, using a container image proxy (harbor), following https://docs.rke2.io/install/airgap#rke2-binary-install and https://docs.rke2.io/advanced#configuring-an-http-proxy:

What I did:

• Debian containerd package is installed but disabled+stopped; no more container related packages installed.
• Environment HTTP(S)_PROXY set to our registry proxy, wich will divert requests to gcr.io, registry-1.docker.io, index.docker.io and some more to our internal Harbor registry proxy. Harbor presents correct certs for all registries, issued by our internal well-known CA.
• selinux is enabled and permissive,
• selinux=true and debug=true in rke2/config.yaml
• Copying rke2 from rke2.linux-amd64.tar.gz to /usr/local/bin (will handle service later).
• Executing /usr/local/bin/rke2 server, which fails:

INFO[0001] Checking local image archives in /var/lib/rancher/rke2/agent/images for index.docker.io/rancher/rke2-runtime:v1.27.6-rke2r1 
WARN[0001] Failed to load runtime image index.docker.io/rancher/rke2-runtime:v1.27.6-rke2r1 from tarball: no local image available for index.docker.io/rancher/rke2-runtime:v1.27.6-rke2r1: not found in any file in /var/lib/rancher/rke2/agent/images: image not found 
INFO[0001] Pulling runtime image index.docker.io/rancher/rke2-runtime:v1.27.6-rke2r1 
WARN[0001] Failed to get image from endpoint: unrecognized challenge:  
FATA[0001] failed to get runtime image index.docker.io/rancher/rke2-runtime:v1.27.6-rke2r1: all endpoints failed: unrecognized challenge:

To test the proxy accessability:

• starting the containerd service, with Environment=HTTP(S)_PROXY added in containerd.service
• ctr image pull index.docker.io/rancher/rke2-runtime:v1.27.6-rke2r1
• The image is pulled successfully, and shows up cached in the harbor project index.docker.io as expected.

So AFAICS there's no problem in the infrastructure present.

What does "unrecognized challenge" mean and how to resolve this issue?

[brandond]

Environment HTTP(S)_PROXY set to our registry proxy, wich will divert requests to gcr.io, registry-1.docker.io, index.docker.io and some more to our internal Harbor registry proxy

A "registry proxy" is not the same thing as a HTTP proxy. A "registry proxy" is a pull-through caching image mirror; you need to configure it as a registry mirror and not just try to use it as the global system HTTP proxy.

See the docs at https://docs.rke2.io/install/containerd_registry_configuration#mirrors, and remove the HTTP_PROXY/HTTPS_PROXY settings from your environment vars.

[andreas-p]

The Harbor REGISTRY PROXY isn't accessed directly, but with an intermediate Nginx HTTP PROXY, so docker/containerd/... pulls don't need to have each upstream repo configured individually as proxied. So this is actually HTTP_PROXY that has to be configured.

As mentioned, this setup works for some years now with rke/docker, and containerd has no problem pulling either. But RKE2 does something different, with no helpful error message.

[brandond]

I've never seen anyone use Harbor that way. The only way I have ever seen it work is when configured as a registry mirror.

If you've figured out how to make harbors pull through registry cache work transparently via a http proxy, please explain how you set that up. With more information I can perhaps suggest why it's not working with containerd.

[andreas-p]

For all developer and cluster machines, docker/containerd only needs to know the HTTP Proxy to access all required repositories, external or internal. Any additional upstream repo just needs central configuration.

Harbor IS configured as registry mirror (as well as our company registry), it's only the centralized configuration with the NGinx proxy intercepting traffic to index.docker.io and diverting to Harbor (and another internal registry).
I'm suspecting that RKE2 does some interaction with the NGINX proxy, that "unrecognized challenge". (There's no auth or other challenge configured in nginx).

[andreas-p]

Investigation traffic at the nginx proxy further, I had the following findings:

• ctr image pull will make two small CONNECTs to index.docker.io, and download the image with a third CONNECT
• rke2 server will make just one small CONNECT and fail.

After reconfiguration of nginx, using the original index.docker.io upstream instead of the Harbor replacement, auth.docker.io is accessed as well, and when enabled the third upstream production.cloudflare.docker.com is contacted.

So when index.docker.io, auth.docker.io and production.cloudflare.docker.com are the original upstreams, rke2 server will work as expected (also pulling from harbor-cached registry-1.docker.io). Apparently, rke2 tries to contact index.docker.io in a way that harbor (latest Version 2.9.0) doesn't support as expected.

[andreas-p]

Digging harbor's log, I can see that rke's request won't get through the nginx proxy without mirror setting, so no URI from there. A working request from containerd will use /v2/index.docker.io/rancher/rke2-runtime:v1.27.6-rke/manifests/... URIs.

Here's the nginx.conf, originally inspired by https://github.com/rpardini/docker-registry-proxy some years ago:

http {
    map_hash_bucket_size 128;
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    gzip  off;

    map $connect_host $interceptedHost {
        hostnames;
        include /etc/nginx/docker.intercept.map; # Exceptions that should bypass Harbor
	default registry.internal.proxy:443;
    }

    # The proxy director layer, listens on 3128
    server {
        listen 3128;
        listen [::]:3128;
        access_log /var/log/nginx/access.log main;

        proxy_connect;
        proxy_connect_address $interceptedHost;
        proxy_max_temp_file_size 0;
	resolver 192.168.0.1 ipv6=off;

        # forward proxy for non-CONNECT request
        location / {
            add_header "Content-type" "text/plain" always;
            return 200 "registry-proxy: The docker registry proxy is working!\n";
        }
    }
}

[brandond]

Huh so its just a dummy forward proxy that sends everything to harbor, instead of what was originally requested. And Harbor is in turn happy to serve requests for docker.io and such? I thought Harbor needed to be set up with specific projects for the pull-through cache - as demonstrated by your rewrite directive to prefix index.docker.io/ to the image reference.

[andreas-p]

Yes, Harbor has projects configured for every upstream to be proxied and cached (and assumed acceptable).

[brandond]

Harbor must look at the host header in the request to identify which project it's supposed to be serving images for? That's the only way I can see this working.

[andreas-p]

Harbor brings its own nginx that does the rewrite.