.buildkit_qemu_emulator: operation not supported

[anthr76]

Discussed in #23890

^{Originally posted by anthr76 September 6, 2024}
Hello! I'm trying to use the Docker CLI with the DOCKER_HOST set to the Podman socket on MacOS.

I have a project I'm trying to build (https://github.com/goauthentik/authentik) that uses buildx.

I'm using a arm64 Mac and the command I'm trying to invoke is

DOCKER_BUILDKIT=1 docker build . --tag authentik:test1 --platform=linux/amd64 -f Dockerfile --load

This fails with a error

/dev/.buildkit_qemu_emulator ... : did not complete successfully: failed to copy xattrs: failed to set xattr "security.selinux" on /tmp/buildkit-qemu-emulator2534885165/dev/.buildkit_qemu_emulator: operation not supported

This does run okay on the native architecture.

Has anyone ever seen this before and know if it's a bug or a workaround?

I found this related issue dagger/dagger#8054

I know disabling SELinux on CoreOS is usually not a route you can really take so not sure how to approach this.

Does anyone know which codepath this lies in?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Sayed-Imran]

@anthr76 did you install the qemu binaries ?

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Try this once

[rgov]

Moving a comment over from #24646:

A partial explanation of what is happening, in my understanding:

1. Podman's machine VM, which runs Fedora CoreOS, ships with SELinux enabled.

2. When using docker buildx build, a BuildKit container is started on the machine VM to build the desired container image. This container provides the emulator buildkit-qemu-x86_64 (say), which is used to execute foreign-architecture binaries during the build process.

3. The buildkit-qemu-x86_64 binary has a security.selinux extended attribute:

    $ docker exec -it buildx_buildkit_default /bin/sh -c 'apk add attr && getfattr -d -m "" -- $(command -v buildkit-qemu-x86_64)'     
    ...
    security.selinux="unconfined_u:object_r:container_ro_file_t:s0"
   

4. When BuildKit prepares the build, it copies buildkit-qemu-x86_64 to a temporary location which is then mounted into the building container (source).

   This copy operation attempts to duplicate all extended attributes. However, SELinux denies this in order to protect the security.selinux access control policy. BuildKit treats the failure to copy the xattr as fatal.

   BuildKit could ignore this error, which was previously done elsewhere in the Docker ecosystem. Copy of buildkit-qemu-emulator should ignore xattr failures moby/buildkit#5544

As a workaround, you can disable SELinux on the machine VM.

I solemnly swear I will not do this in production.

1. Enter the machine VM with podman machine ssh and edit /etc/sysconfig/selinux to set SELINUX=disabled.

2. Restart the machine VM with podman machine stop && podman machine start.

3. Ensure there are no lingering BuildKit containers -- existing ones won't work after SELinux policy has changed. Clear them out with docker ps -a and docker rm -f <id>.

4. Clear the Docker build cache for good measure: docker buildx prune