How to use local registry in a CI setup (DinD)

[MShekow]

Hi,

I'm wondering how you use k3d in a CI setup. Here it is common that you build your own images first, and then they need to be accessible from within the cluster. I'd like to use the local registry for this (k3d cluster create mycluster --registry-create).

However, I have not found a way to even push the images into the local registry. I'm currently still experimenting on Windows (Docker for Windows), I hope that this is not the issue here.

On the host, I'm running docker run --rm -it --name mycontainer -v "//var/run/docker.sock:/var/run/docker.sock" myimage /myscript.sh

Inside the mycontainer container, I create the k3d cluster (k3d cluster create mycluster --registry-create) and then I attempt to push images into its local registry. However, both approaches I tried don't seem to work:

Approach 1) run docker network connect k3d-mycluster mycontainer (as otherwise k3d-mycluster-registry would not even be resolvable), then retag the image, e.g. docker tag myimage:latest k3d-mycluster-registry:5000/myimage:latest and then try to push it: docker push k3d-mycluster-registry:5000/myimage:latest

Returned error: Get https://k3d-mycluster-registry:5000/v2/: dial tcp: lookup k3d-mycluster-registry on 192.168.65.5:53: read udp 192.168.65.4:43967-\u003e192.168.65.5:53: i/o timeout

Approach 2) use the gateway IP (and the host port): for this I determine the Gateway IP of mycontainer and then I run something like docker tag myimage:latest <gateway-ip>:34567/myimage:latest (where 34567 is the registry's port on the host). The corresponding docker push command then fails with this error:

Get https://<gateway-ip>:34567/v2/: http: server gave HTTP response to HTTPS client

I'm not sure how to continue.

[iwilltry42]

Hi @MShekow , thanks for starting this discussion!
I assume you checked this page already? https://k3d.io/usage/guides/registries/#pushing-to-your-local-registry-address

So let's see if I understand your setup correctly:

Docker for Windows -> docker-in-docker (host docker socket mounted) -> k3d

Now you want to push an image to the k3d managed registry from within that dind container, right?

Returned error: Get https://k3d-mycluster-registry:5000/v2/: dial tcp: lookup k3d-mycluster-registry on 192.168.65.5:53: read udp 192.168.65.4:43967-\u003e192.168.65.5:53: i/o timeout

I'm not 100% sure here... but this looks more like the lookup of the name is failing, right?
What address is 192.168.65.5? Is this a docker-owned address?

Get https://<gateway-ip>:34567/v2/: http: server gave HTTP response to HTTPS client

This may be getting you somewhere: the registry does not run with HTTPS, but docker expects (and wants https).
Usually, when k3d runs on your localhost, docker is fine with that, as 127.0.0.1 is on the "safelist" for insecure registry connections.
But with your setup, the registry is "somewhere else", so you'd have to add it to the list of insecure registries, see e.g. https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry or https://docs.docker.com/engine/reference/commandline/dockerd/#insecure-registries

[iwilltry42]

There seem to be quite a few specific settings playing together to make this especially hard to achieve 🤔
What about k3d image import as another alternative?

[MShekow]

k3d image import is not feasible in this specific case. I'm building the K8s version of an existing Dockerized software, built by another team. They decided to only use the :latest version tag and therefore, imagePullPolicy is set to Always (this is implicit Kubernetes behavior, which also makes sense, as we do want to check whether there is a newer version in the registry than the one cached on the worker node, and pull it if this is the case). But that means that image are pulled, and that requires a registry (to my understanding).

[dtomasi]

Isn´t it much easier to use the container registry your self hosted gitlab provides? To pull images to your k3d cluster you only have to provide a secret with a gitlab access token.
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
https://docs.gitlab.com/ee/user/packages/container_registry/

This makes it also possible to use the same image on your local machine and in gitlab CI without the need of retagging it to k3d-registry:5000 .... or so ... and push it again to your local registry.

[MShekow]

@dtomasi in principle you are right. So far, my train of thought was that a CI/CD pipeline works as follows:

• Build the current software and package it as Docker images (but don't push them yet)
• Run the CI test (including k3d and pushing the locally-built images into its local registry)
• Only if the CI test passed, we actually push the images into GitLab's OCI registry (as otherwise it would be a waste of time and resources to push images which are most likely broken anyway)

Of course there are other approaches. E.g. having separate OCI registries, e.g. a "production" registry with working images, and a development/CI registry using GitLab's own registry, that contains only testing images (and has aggressive garbage collection settings).

[dtomasi]

There are multiple ways to speed up and save resources ...

• Use Multi stage docker builds with proper caching
• Use buildx with cache from a previous build image to get cached layers.
• If a you do so, only the top level layers containing your software have to be pushed to the registry because the OS or package layers already exists.

I would recommend to push the images to Gitlab Container registry using for example the short Commit SHA as a tag and have a cleanup job that deletes images when tests are failing or simply just use Gitlab cleanup policy, which removes images that match a pattern periodically.
https://docs.gitlab.com/ee/user/packages/container_registry/#delete-images-by-using-a-cleanup-policy