How to use custom k3s image

[leelavg]

• Followed this guide
• Create image from below Dockerfile

version=$(curl -s https://update.k3s.io/v1-release/channels | jq -r '.data[]|select(.id=="latest")|.latest')
version=${version/+/-} # v1.22.4-k3s1

FROM rancher/k3s:$version as k3s
FROM alpine:3
RUN apk add --no-cache util-linux udev lvm2 gptfdisk sgdisk
COPY --from=k3s / /
RUN mkdir -p /etc && \
    echo 'hosts: files dns' > /etc/nsswitch.conf && \
    echo "PRETTY_NAME=\"K3s ${version}\"" > /etc/os-release && \
    chmod 1777 /tmp
VOLUME /var/lib/kubelet
VOLUME /var/lib/rancher/k3s
VOLUME /var/lib/cni
VOLUME /var/log
ENV CRI_CONFIG_FILE="/var/lib/rancher/k3s/agent/etc/crictl.yaml"
ENTRYPOINT ["/bin/k3s"]
CMD ["agent"]

# k3d version
k3d version v5.1.0
k3s version v1.21.5-k3s2 (default)

• Able to create k3d cluster with default image however not able to create with above image
• Attaching the o/p of k3d cluster create test -a 0 --image=leelavg/k3s:v1.22.4-k3s1 --verbose
• k3d.log
• Error log

ERRO[0003] Failed Cluster Start: Failed to start server k3d-test-server-0: Node k3d-test-server-0 failed to get ready: error waiting for log line `k3s is up and running` from node 'k3d-test-server-0': stopped returning log lines 
ERRO[0003] Failed to create cluster >>> Rolling Back    
FATA[0003] Cluster creation FAILED, all changes have been rolled back!

• Am I making any mistake here?

[iwilltry42]

Hi @leelavg , thanks for starting this discussion!
Can you please provide some logs of the container?
i.e. start the cluster with k3d cluster create test -a 0 --image=leelavg/k3s:v1.22.4-k3s1 --verbose --no-rollback and wait for it to fail, then copy the output of docker ps -a and docker logs k3d-test-server-0

[leelavg]

@iwilltry42

• yes, above is the issue, I was using this script to create custom k3s image and it worked after removing []
• now I hit another issue, and this is server log k3d.log
• I can see good amount of space but somewhere mounts/volumes aren't working as expected

-> df -hT /tmp
Filesystem     Type   Size  Used Avail Use% Mounted on
tmpfs          tmpfs  7.4G   49M  7.3G   1% /tmp

-> k3d version
k3d version v5.1.0
k3s version v1.21.5-k3s2 (default)

• This is the cli command used to create cluster
• Is it due to /var/lib/kubelet internally being a docker volume and shared mounts aren't working 🤔

[iwilltry42]

Now this is a pretty complex setup right there O.o
I guess that you're right with your assumptions about the mounts being the issue. Why are you even mounting the pods dir from tmp? 🤔
Anyway, as this is not a problem with k3d anymore but with some advanced usage of custom K3s images, I'm not sure I can help you more on this 🤔

[leelavg]

Why are you even mounting the pods dir from tmp

It's due to #206 (comment)

complex setup right there

IMHO, it isn't a complex setup, it's just there are multiple volume mounts and I'm not doing HA/magic network stuff 😅

I'm not sure I can help you more on this

ack, thanks for the help so far, I don't want to move from k3d due to this usecase, will invest sometime on this and will pick-up minikube with libvirt if things aren't working.

[leelavg]

@iwilltry42

• not sure what changed/happened, after a reboot the cluster creation is all good and I don't see above errors in the logs

-> dp
CONTAINER ID   IMAGE                      COMMAND                  CREATED              STATUS              PORTS                                       NAMES
f898caa849bf   rancher/k3d-tools:5.1.0    "/app/k3d-tools noop"    About a minute ago   Up About a minute                                               k3d-test-tools
8e331960ab0d   rancher/k3d-proxy:5.1.0    "/bin/sh -c nginx-pr…"   2 minutes ago        Up About a minute   80/tcp, 0.0.0.0:36447->6443/tcp             k3d-test-serverlb
3151131a8a00   leelavg/k3s:v1.22.4-k3s1   "/bin/k3d-entrypoint…"   2 minutes ago        Up About a minute                                               k3d-test-agent-2
77def3d8332d   leelavg/k3s:v1.22.4-k3s1   "/bin/k3d-entrypoint…"   2 minutes ago        Up About a minute                                               k3d-test-agent-1
b74de2c49228   leelavg/k3s:v1.22.4-k3s1   "/bin/k3d-entrypoint…"   2 minutes ago        Up About a minute                                               k3d-test-agent-0
0ae7060f8c80   leelavg/k3s:v1.22.4-k3s1   "/bin/k3d-entrypoint…"   2 minutes ago        Up 2 minutes                                                    k3d-test-server-0

• pods are scheduled and running

-> k get po -n kube-system
NAME                                READY   STATUS      RESTARTS        AGE
metrics-server-9cf544f65-jbgkh      1/1     Running     0               3m29s
coredns-85cb69466-pb7xl             1/1     Running     0               3m29s
helm-install-traefik-crd--1-f49fk   0/1     Completed   0               3m29s
helm-install-traefik--1-88bk7       0/1     Completed   1               3m29s
svclb-traefik-4qzdt                 2/2     Running     1 (2m41s ago)   2m53s
svclb-traefik-8prkh                 2/2     Running     1               2m53s
svclb-traefik-zqhq9                 2/2     Running     4 (2m38s ago)   2m53s
traefik-786ff64748-d5k9s            1/1     Running     0               2m53s
svclb-traefik-pmwss                 2/2     Running     3 (2m24s ago)   2m53s

• on to the last issue, if I'm not able to make it work, I should stop investing time
• for ex. while creating the k3d cluster I mapped /dev/sdq to agent-2 but could see I can perform read/write of that disk from server-0 as well
• might it be due to running k3d node as privileged or anything that you heard of earlier along the similar lines?

[iwilltry42]

might it be due to running k3d node as privileged or anything that you heard of earlier along the similar lines?

Yep, you can just try to docker exec k3d-k3s-default-server-0 sh -c "ls /dev" in the default k3d cluster, you see that all devices are present there. (Same as docker run --rm -it alpine sh -c "ls /dev" vs docker run --rm --privileged -it alpine sh -c "ls /dev")

[leelavg]

reply moved to #874 (reply in thread) by @iwilltry42