Alpine docker-host challenge

[krzysztof-abramowicz]

I chose Lima to build my custom, light & fast Docker Desktop alternative with Alpine Linux as my docker-host VM’s OS (vm-host: macOS 15.3 @ M2). It seemed to be a pleasing warmup before my in-depth Docker learning and further experiments. Unfortunately, I encountered several irritating problems:

1. Starting an Alpine Linux machine hangs on [hostagent] Waiting for the essential requirement 2 of 4: "user session is ready for ssh" (for QUEMU, “2 of 2” for VZ). Despite of the error, machine becomes available, but it ruins the user experience. My fix: Adding a provisioning script (provision: section) seems to help, but why?
2. The created user does not belong to the wheel group by default, so one cannot execute commands with elevated privileges. My fix: the group membership may be added manually in the provisioning script.
3. The created user doesn’t have a password and this cannot be changed with passwd as there’s no password. Minor issue. My fix: sudo passwd <user> does not prompt for the current, non-existent password.
4. The created user does not belong to the docker group, thus cannot connect to the docker daemon (access it's socket). This cannot be fixed by the provisioning script as by its execution the user session has already been started; the association, although visible in /etc/group, doesn’t take effect. Manual fix: sudo reboot
5. The user-space x86_64 emulation doesn’t work. I’m getting exec format error on both QEMU and VZ machine. My fix: The qemu-x86_64 and qemu-openrc packages may be installed, and the qemu-binfmt service started manually (thus the binary format is registered in the kernel).
6. The Rosetta share doesn’t seem to be exposed to the VM. I’d try to mount it manually, but I can’t see other devices than virtio2 (vda, local mounts) and virtio3 (vdb, cidata). How can I check if Lima has exposed the Rosetta share to the VM correctly? May I choose to use Rosetta or QUEMU as amd64 handler through the configuration?
7. The log item tcpproxy: for incoming conn 127.0.0.1:50635, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: connection was refused bypasses JSON formatting, thus the ha.stderr.log cannot by parsed with jq without additional filtering. Small thing, but annoying.
8. There are plenty of Stopping udp proxy entries in the log, and they take up most of it over time! Does it have to happen so frequently? And what's the reason for it?

As you can see, I found a way to finally got close to my dream result, but I believe it could be achieved much easier. Maybe I haven't understand some design assumptions, or maybe a few things could be improved in Lima? I'm even eager to contribute some solutions myself, but I have to become a successful power user first :)

I'm also aware of the fact, that there are docker-enabled VM templates build on Ubuntu, but I got used to using Alpine in the cloud and I'd like to experiment with it on my desktop as well. Besides, It's Linux – everything is possible! Right?

I will be grateful for any tips, improvements or suggestions for my possible contributions.

My current VM configuration:

images:
- location: "https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/nocloud_alpine-3.21.2-x86_64-uefi-cloudinit-r0.qcow2"
  arch: "x86_64"
  digest: "sha512:1aaf22b4a584e69e228e6aa38a295159c0143d9ccebe7ad4928e92b414714066af3bfe5f9e0ca4d4d64a70ca9fea09033af90258a6f2344130d70b660151127a"
- location: "https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/nocloud_alpine-3.21.2-aarch64-uefi-cloudinit-r0.qcow2"
  arch: "aarch64"
  digest: "sha512:08d340126b222abae651a20aa63c3ee3dc601d703de7879d2a6bc1fe82a3664d058a2c55ad0cf8a874327f7535e3af8a9384ce438217d6f32200cad1462a5b32"

vmType: vz
rosetta:
  enabled: true
  binfmt: true

mountType: virtiofs
mounts:
- location: "~"
- location: "/tmp/lima"
  writable: true

containerd:
  system: false
  user: false

provision:
- mode: system
  script: |
    # 1. sudo  
    apk add sudo
    adduser krab wheel
    # 2. Docker daemon
    apk add docker
    adduser krab docker
    service docker start
    rc-update add docker default
    # 3. QEMU user-space amd64 emulation
    apk add qemu-x86_64 qemu-openrc
    rc-service qemu-binfmt start
    rc-update add qemu-binfmt default

portForwards:
- guestSocket: "/var/run/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"

[afbjorklund]

It is probably easier to start with Lima's Alpine as the base, and it has a configuration with docker as well:

https://github.com/lima-vm/alpine-lima

Here is such an "edition" that I made before:

• Create edition with docker installed alpine-lima#45

Basically it bundles the needed apk with the ISO

But in the end it wasn't really worth it, since you could just apk add docker to the regular alpine OS.

• Make editions that install the regular alpine packages alpine-lima#107

• Add Alpine examples with the container engines #1619

---

tl;dr limactl start template://alpine

[afbjorklund]

Another interesting project is TinyCore Linux, which was the basis for docker-machine/boot2docker and podman-machine*
* version 1, before Red Hat wrote a new version 3 based on Fedora CoreOS: https://github.com/boot2podman/boot2podman

[krzysztof-abramowicz]

I think that it would be great to have control over users and groups through YAML files directly. Especially, as the provisioning script runs while the user session has already been started (why, actually?). It could also allow to change the default user used for shell access.

[afbjorklund]

Adding groups should be easy (in cloud-init), but the downside was that it "stole" UID 1000 which looked slightly odd later on

• Add configuration option to add user to one or more groups. #1686 (comment)

As a workaround, we made the Lima user own the rootful socket instead of using a docker/podman group for the same thing.

The preferred method is rootless containers, but it doesn't work so well on a non-standard system like Alpine Linux...

[afbjorklund]

It could also allow to change the default user used for shell access.

The user is the same, that is part of the illusion we are maintaining. (Wizard of Oz mode: "pay no attention to the VM ....")

[afbjorklund]

There is "plain" mode now, but some users requested a more vagrant-like mode ("lima" user, share $PWD instead of $HOME)

I don't think we added any GitHub issue about it, though? Just some discussions, after Vagrant stopped being Open Source.

[afbjorklund]

I can't reproduce the first two items, so you might want to open a bug report if it does not work for you.

1. The user session should be there in /var/log/cloud-init-output.log

+ cp /mnt/lima-cidata/meta-data /run/lima-ssh-ready

2. The user should be added to /etc/sudoers.d/90-cloud-init-users

anders ALL=(ALL) NOPASSWD:ALL

The other ones seem to be more known.

[afbjorklund]

Might have to add that to the bootstrapping for Alpine?

sudo modprobe binfmt_misc

sudo mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc

Otherwise it will be hard to mount and register Rosetta.

---

But it should have been done by the packages:

if [ -f /etc/alpine-release ]; then
        rc-service procfs start --ifnotstarted
        rc-service qemu-binfmt stop --ifstarted
fi

Except for: "* rc-service: service `qemu-binfmt' does not exist"

[afbjorklund]

• The alpine image needs to install qemu-openrc, for the qemu-binfmt service #3278

[afbjorklund]

After installing the package and restarting the instance, the mount is there and it is registered too:

vz-rosetta on /mnt/lima-rosetta type virtiofs (rw,relatime)

/proc/sys/fs/binfmt_misc/rosetta

If you installed qemu-user (like on qemu-system), there's also /proc/sys/fs/binfmt_misc/qemu-x86_64

[krzysztof-abramowicz]

Thanks for another great dose of knowledge! It will take me a while to absorb it and analyse all the mentioned issues. My observations so far look as follows (while still working with the template initially provided):

• [1] The /var/log/cloud-init-output.log mentions Deprecated cloud-config provided: users.0.ssh-authorized-keys and just below the Running module ca_certs (...) failed. I have no idea if it slows down the SSH initialisation but it might be worth updating anyway. There are a few other warnings:
  • Could not find module named cc_reset_rmc,
  • Failed to execute /mnt/lima-cidata/boot/05-rosetta-volume.sh (!),
  • Failed to run module scripts_per_boot.
• [2] The /etc/sudoers.d/90-cloud-init-users is created by the cloud-init as expected. But I still had to add myself to the wheel group and install the sudo package to make it work.
• [5] With my configuration, binfmt_misc kernel module is already available, but, as you mentioned , it's because the qemu-binfmt service is running. But I added it manually in the provisioning script. And yes, I can see* rc-service: service 'qemu-binfmt' does not exist in the cloud-init log... even after restarting the VM. However, I guess, it didn't help as:
  • the provisioning script executes after cloud-init,
  • the services at run level default start after cloud-init too,
  • cloud-init runs only once for an instance either way.
• So the only solution I see so far is to re-run cloud-init after either:
  • adding qemu-binfmt to an "earlier" run level, if any is "early" enough here;
  • modifying cloud-init configuration from the provisioning script, if already available.
• Or, of course, prepare a custom VM image with everything needed already on board!
• [6] As for the Rosetta itself, it seems that I've got it mounted correctly either way. Maybe a restart helped somehow. Does it mean that 05-rosetta-volume.sh is a boot script added by cloud-init, not a cloud-init script? And the log refers to the first VM boot only, so I shouldn't care? But the remaining step is to register it as binary handler, still. Apple's documentation presents a low-level approach, but I believe, as always, that there is an easier way to do it!

[jandubois]

• [1] The /var/log/cloud-init-output.log mentions Deprecated cloud-config provided: users.0.ssh-authorized-keys and just below the Running module ca_certs (...) failed. I have no idea if it slows down the SSH initialisation but it might be worth updating anyway. There are a few other warnings:

The deprecation messages are normal. I've explained in #2267 (comment) why we can't just update to the latest.

If you get cloud-init failures, then please create an issue and attach both your lima.yaml file and the /var/log/cloud-init-output.log file.

• Could not find module named cc_reset_rmc,
• Failed to execute /mnt/lima-cidata/boot/05-rosetta-volume.sh (!),

Are you sure you are running a Lima compiled from HEAD of master. This was a bug that we just fixed a couple of days ago.

• Failed to run module scripts_per_boot.

That's probably a consequence of failing the 05-rosetta-volume.sh script.

• [2] The /etc/sudoers.d/90-cloud-init-users is created by the cloud-init as expected. But I still had to add myself to the wheel group and install the sudo package to make it work.

This should be done by our builtin provisioning scripts; sudo is installed by 10-alpine-prep.sh. Please build Lima from HEAD of master, and if it still fails, send the cloud-init log file.

I can't follow the rest of your comments, but it sounds like you make it much too complicated.

I've shown you yesterday how you can create an Alpine instance from the commandline with Rosetta installed and ready to use.

You should be able to combine this with the commands to add docker and start it, and have a working docker engine running on Alpine.

[jandubois]

provision:
- mode: system
  script: |
    # 1. sudo  
    apk add sudo
    adduser krab wheel

You can use template expressions in provisioning scripts, so it might be good to use {{.User}} in case you want to share your template with others.

    # 2. Docker daemon
    apk add docker
    adduser krab docker
    service docker start
    rc-update add docker default

I don't think the rc-update is needed because the provisioning script is executed on every boot, not just the first one.

    # 3. QEMU user-space amd64 emulation
    apk add qemu-x86_64 qemu-openrc
    rc-service qemu-binfmt start
    rc-update add qemu-binfmt default

This was a bug in Lima that has been fixed in #3279. You no longer need to install qemu-binfmt if you want to use Rosetta (using Rosetta will just disable it again).

[krzysztof-abramowicz]

1. Great point with {{.User}} – I wasn't sure if I can use template expressions in the script too.
2. Great to know before trying to add reboot after adding user to the docker group!
3. This was intended as an intermediate solution, before even trying to make it work with Rosetta.

Also, I'd like to try both QEMU and Rosetta emulation for amd64 and make this configurable somehow through the template, my Makefile or this provisioning script – that's still one of my todos. For now, I have a working QEMU version and I'd like to know how to manually check if the Rosetta share was created correctly and is ready to be mounted. I cannot see it with lsblk and there's no VZ CLI tool to check the host side, is it?

[jandubois]

I'd like to know how to manually check if the Rosetta share was created correctly and is ready to be mounted.

Lima already mounts it for you, so you can just check the content of /mnt/lima-rosetta:

❯ limactl start --tty=false --set '.rosetta={"enabled":true,"binfmt":true}' template://alpine
...
❯ limactl shell alpine ls /mnt/lima-rosetta
rosetta   rosettad

I'd like to try both QEMU and Rosetta emulation for amd64 and make this configurable somehow through the template

I would suggest using the param mechanism for this, to make it easily configurable from the commandline.

[jandubois]

I would suggest using the param mechanism for this, to make it easily configurable from the commandline.

I just realized that this is currently not possible because rosetta.* cannot be set via template variables right now. I'm afraid you will need to use 2 different template files for it.

Or you use --set .rosetta.enabled=true for selecting rosetta, and then you install and configure qemu-binfmt if /mnt/lima-rosetta does not exist.

BTW, in case this wasn't clear: you need to build Lima from the latest master to have Rosetta working with the template://alpine to get the bug fix.