In OCP 4, starting with buildah:v1.32.2, buildah bud command fails its execution when secret etc-pki-entitlement is mounted on /etc/pki/entitlement

[gmeghnag]

ISSUE

In OCP 4 when I execute a buildah pod running quay.io/buildah/stable:v1.32.2 (or upper version) and mounting etc-pki-entitlement as a secret the pod fails its execution with the following error:

time="2024-10-29T13:55:57Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah3386043866/mnt\": directory not empty"
time="2024-10-29T13:55:57Z" level=debug msg="error cleaning up intermediate mount NS: remounting \"/var/tmp/buildah3386043866/mnt/rootfs/etc/pki/entitlement\" in mount namespace with flags 0x0 instead of 0x1: permission denied"

The same does not happens if we use image quay.io/buildah/stable:v1.32.0 (or lower).

HOW REPRODUCIBLE

100 %

REPROUCER

You can find a simple reproducer here; basically, the pod that would trigger the error is the following:

apiVersion: v1
kind: Pod
metadata:
  name: buildah-issue-5800-1-32-2-not-working
spec:
  volumes:
  - name: etc-pki-entitlement
    secret:
      secretName: etc-pki-entitlement
  restartPolicy: Never
  containers:
  - name: buildah-1-32-2-not-working
    image: quay.io/buildah/stable:v1.32.2
    volumeMounts:
      - name: etc-pki-entitlement
        mountPath: /etc/pki/entitlement
        readOnly: false 
    command:
    - /bin/sh
    - -c
    - |
      #!/bin/bash
      mkdir -p rootfs/etc/pki/rpm-gpg
      mkdir -p rootfs/etc/yum.repos.d
      curl -sk https://raw.githubusercontent.com/gmeghnag/buildah-issue-5800/refs/heads/main/rootfs/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official > rootfs/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official
      curl -sk https://raw.githubusercontent.com/gmeghnag/buildah-issue-5800/refs/heads/main/rootfs/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-9 > rootfs/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-9
      curl -sk https://raw.githubusercontent.com/gmeghnag/buildah-issue-5800/refs/heads/main/rootfs/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release > rootfs/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
      curl -sk https://raw.githubusercontent.com/gmeghnag/buildah-issue-5800/refs/heads/main/rootfs/etc/yum.repos.d/centos9.repo > rootfs/etc/yum.repos.d/centos9.repo
      curl -sk https://raw.githubusercontent.com/gmeghnag/buildah-issue-5800/refs/heads/main/rootfs/etc/yum.repos.d/epel.repo > rootfs/etc/yum.repos.d/epel.repo
      curl -sk https://raw.githubusercontent.com/gmeghnag/buildah-issue-5800/refs/heads/main/rootfs/etc/yum.repos.d/redhat9.repo > rootfs/etc/yum.repos.d/redhat9.repo
      curl -sk https://raw.githubusercontent.com/gmeghnag/buildah-issue-5800/refs/heads/main/Dockerfile > Dockerfile
      buildah bud --log-level debug --storage-driver vfs -v /etc/pki/entitlement:/etc/pki/entitlement/ -t test-image:latest .
    securityContext:
      capabilities:
        add:
        - SETFCAP

[nalind]

Do you have the full log output?
The remounting message indicates that it failed to change the mount flags on the bind mount from read-only to read-write. If the /etc/pki/entitlement mount in the build container ("buildah-1-32-2-not-working") is read-only, as the error message suggests, try adding the :ro option to the -v flag argument.

[gmeghnag]

Do you have the full log output? The remounting message indicates that it failed to change the mount flags on the bind mount from read-only to read-write. If the /etc/pki/entitlement mount in the build container ("buildah-1-32-2-not-working") is read-only, as the error message suggests, try adding the :ro option to the -v flag argument.

@nalind modifying the buildah bud command as follow did the trick!

buildah bud --log-level debug --storage-driver vfs -v /etc/pki/entitlement:/etc/pki/entitlement/:ro -t test-image:latest .

Should we investigate why it was not necessary in version v1.32.0 and lower?

Thank you!

[nalind]

My suspicion is that when parsing the bind mount options, we make the implicit "rw" explicit, and the logic which later ensures that the flags we have at the target mount match the desired flags trips over it.