docker buildx build fails for foreign architecture

[rgov]

Issue Description

I'm trying to build a container image for a foreign architecture using alias docker=podman. The command is:

docker buildx build . --platform linux/amd64

The build fails with:

"/dev/.buildkit_qemu_emulator /bin/sh -c apt update ..." did not complete successfully:
failed to copy xattrs: failed to set xattr "security.selinux" on
/tmp/buildkit-qemu-emulator193598978/dev/.buildkit_qemu_emulator:
operation not supported

I don't know if this is an issue with Podman per se, as the issue is also filed against Docker. However, I am filing it here because I am using a podman machine (on a macOS host) and so it may be that a machine VM configuration issue.

I attempted to disable SELinux on the machine VM with sudo setenforce 0 but this didn't resolve the issue. Using a rootful machine did not work either.

podman info output

`podman info` output

host:
  arch: arm64
  buildahVersion: 1.37.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.93
    systemPercent: 0.05
    userPercent: 0.02
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2032
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.10.10-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 938639360
  memTotal: 2043760640
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.20241007140227477357.main.38.g08fbf82.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.0-dev
    package: netavark-1.12.1-1.20241007131025236895.main.62.g47632d8.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.0-dev
  ociRuntime:
    name: crun
    package: crun-1.17-1.20241007140634150540.main.7.g7c194cb.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 4f2c23486977b381fd9461150d2c0038b7d918b3
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.aarch64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 2h 49m 1.00s (Approximately 0.08 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 15
    paused: 0
    running: 1
    stopped: 14
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 28086169600
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 132
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.4
  Built: 1728259200
  BuiltTime: Sun Oct  6 17:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/arm64
  Version: 5.2.4

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

[rgov]

I've filed moby/buildkit#5544 for a possible fix in BuildKit.

[rhatdan]

Looks like something is attempting to set SELinux labels on a file system that does not support it.

[rgov]

A partial explanation of what is happening, in my understanding:

1. Podman's machine VM, which runs Fedora CoreOS, ships with SELinux enabled.

2. When using docker buildx build, a BuildKit container is started on the machine VM to build the desired container image. This container provides the emulator buildkit-qemu-x86_64 (say), which is used to execute foreign-architecture binaries during the build process.

3. The buildkit-qemu-x86_64 binary has a security.selinux extended attribute:

    $ docker exec -it buildx_buildkit_default /bin/sh -c 'apk add attr && getfattr -d -m "" -- $(command -v buildkit-qemu-x86_64)'     
    ...
    security.selinux="unconfined_u:object_r:container_ro_file_t:s0"
   

4. When BuildKit prepares the build, it copies buildkit-qemu-x86_64 to a temporary location which is then mounted into the building container (source).

   This copy operation attempts to duplicate all extended attributes. However, SELinux denies this in order to protect the security.selinux access control policy. BuildKit treats the failure to copy the xattr as fatal.

   BuildKit could ignore this error, which was previously done elsewhere in the Docker ecosystem. Copy of buildkit-qemu-emulator should ignore xattr failures moby/buildkit#5544

As a workaround, you can disable SELinux on the machine VM.

I solemnly swear I will not do this in production.

1. Enter the machine VM with podman machine ssh and edit /etc/sysconfig/selinux to set SELINUX=disabled.

2. Restart the machine VM with podman machine stop && podman machine start.

3. Ensure there are no lingering BuildKit containers -- existing ones won't work after SELinux policy has changed. Clear them out with docker ps -a and docker rm -f <id>.

4. Clear the Docker build cache for good measure: docker buildx prune

[Luap99]

Duplicate of #24000