GCP registry returns 404 on digest check

[reggermont]

Error looking very similar to #153

Context

Following the official ingress-nginx values.yaml:

  # ...
  #
  # -- Annotations to be added to the controller Deployment or DaemonSet
  ##
  annotations: {}
  #  keel.sh/pollSchedule: "@every 60m"

  # -- Labels to be added to the controller Deployment or DaemonSet and other resources that do not have option to specify labels
  ##
  labels: {}
  #  keel.sh/policy: patch
  #  keel.sh/trigger: poll
  #
  # ...

and uncommenting the keel configurations, keel detects it and tries to check latest ingress-nginx image digest, but fails

2024-06-07T16:41:58.988016535Z time="2024-06-07T16:41:58Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Head \"https://europe-west9-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c\": http: non-successful response (status=404 body=\"\")" image="ingress-nginx/controller@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c" password= username=

2024-06-07T16:41:58.988060225Z time="2024-06-07T16:41:58Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Head \"https://europe-west9-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c\": http: non-successful response (status=404 body=\"\")" image="namespace:ingress-nginx,image:registry.k8s.io/ingress-nginx/controller:sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c,provider:kubernetes,trigger:poll,sched:0 */3 * * * *,secrets:[]"

2024-06-07T16:41:58.988068955Z time="2024-06-07T16:41:58Z" level=error msg="trigger.poll.manager: got error(-s) while watching images" error="encountered errors while adding images: Head \"https://europe-west9-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c\": http: non-successful response (status=404 body=\"\")"

According to the logs, keel fetch GCP with a head request and receive a 404. Therefore, running a curl from the keel pod itself returns a 200

apk add curl && curl -v -I https://europe-west9-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c

Logs

/ # apk add curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/7) Installing brotli-libs (1.1.0-r1)
(2/7) Installing c-ares (1.27.0-r0)
(3/7) Installing libunistring (1.1-r2)
(4/7) Installing libidn2 (2.3.4-r4)
(5/7) Installing nghttp2-libs (1.58.0-r0)
(6/7) Installing libcurl (8.5.0-r0)
(7/7) Installing curl (8.5.0-r0)
Executing busybox-1.36.1-r15.trigger
OK: 12 MiB in 23 packages
/ # curl -v -I https://europe-west9-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
* Host europe-west9-docker.pkg.dev:443 was resolved.
* IPv6: 2a00:1450:400c:c0a::52
* IPv4: 173.194.76.82
*   Trying 173.194.76.82:443...
* Connected to europe-west9-docker.pkg.dev (173.194.76.82) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.googlecode.com
*  start date: Jun  3 07:31:43 2024 GMT
*  expire date: Aug 26 07:31:42 2024 GMT
*  subjectAltName: host "europe-west9-docker.pkg.dev" matched cert's "*.pkg.dev"
*  issuer: C=US; O=Google Trust Services; CN=WR2
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha384WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://europe-west9-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: europe-west9-docker.pkg.dev]
* [HTTP/2] [1] [:path: /v2/k8s-artifacts-prod/images/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> HEAD /v2/k8s-artifacts-prod/images/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c HTTP/2
> Host: europe-west9-docker.pkg.dev
> User-Agent: curl/8.5.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
HTTP/2 200 
< content-length: 990
content-length: 990
< content-type: application/vnd.docker.distribution.manifest.list.v2+json
content-type: application/vnd.docker.distribution.manifest.list.v2+json
< docker-content-digest: sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
docker-content-digest: sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
< docker-distribution-api-version: registry/2.0
docker-distribution-api-version: registry/2.0
< date: Fri, 21 Jun 2024 14:22:53 GMT
date: Fri, 21 Jun 2024 14:22:53 GMT
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

< 
* Connection #0 to host europe-west9-docker.pkg.dev left intact
/ # 

Workaround

WARNING: Security hole here, use it at your own risk

In ingress-nginx values.yaml, nullify the digest:

controller:
  image:
    digest: null
    digestChroot: null

Keel doesn't check image digest and update works