Reproducible container image builds

[apyrgio]

On build reproducibility

Our Dangerzone container image is currently not reproducible. This means that if we build the container image now and someone rebuilds it tomorrow, the result will probably be different. Why is this important? The concept of reproducible builds and their importance is explained in https://reproducible-builds.org/, but there is a specific reason why it's important for our project.

Currently, we build and sign the Dangerzone container image in machines that we own, because we don't trust third-party servers. A security hole in GitHub, or in our CI pipeline (check out the Ultralytics cryptominer saga), may allow attackers to plant a malicious image with no detection. There are a few drawbacks with this approach though:

• Third parties cannot easily audit if the container image has been built correctly or - gasp - if it has been tampered with. For instance, our Apple Silicon image builds PyMuPDF from source, and while the PyPI source package is hashed, the produced output does not have a known hash. So, it's not easy to verify it's been built correctly (read also the seminal "Reflections on Trusting Trust" lecture by Ken Thompson on that subject).
• We still need to trust something, and in this case it's the FPF-provisioned hardware, where we build the image. It's a better situation for us devs to trust the hardware we manage, sure, but it also paints a huge target to these machines, and ourselves.
• We cannot use caching safely to speed up repeated builds, since stale packages may remain in the image from an older build.

On the other hand, reproducibility opens the door for building the image in third-party servers. If the image has provenance information, e.g., the commit it was built from, then third-parties can rebuild it locally, and check that the contents are exactly the same byte-for-byte. For an application of this, check out #1006.

Current limitations

We've stated that our current container image is not reproducible, but we haven't explained why. There are two main reasons why this is the case:

• We build PyMuPDF from source, because it's not available in Alpine Linux. The result of this build is not reproducible. Note that PyMuPDF wheels are available from PyPI, but there are no ARM wheels for musl libc platforms.
• Alpine Linux does not have a way to pin packages and their dependencies, and does not retain old packages. There's a workaround to download the required packages and store them elsewhere, but then the cached package downloads cannot be easily audited.

[apyrgio]

Proposal

We can take advantage of the Debian snapshot archives and pin our packages by specifying a date. There's already prior art for that, thanks to the incredible work of @AkihiroSuda on reproducible containers. As for PyMuPDF, it is available from the Debian repos, so we won't have to build it from source.

Here are a few other obstacles that we can overcome:

• We currently download the latest gVisor version from a GCS bucket. Now that we have switched to Debian, we can take advantage of their timestamped APT repos and download specific releases from those. An extra benefit is that such releases are signed with their APT key.

• We can no longer update the packages in the container image by rebuilding it. We have to bump the dates in the Dockerfile first, which is a minor hassle, but much more declarative.

• The repro-source-list-.sh script uses the release date of the container image. However, the Debian image is not updated daily (see newest tags in DockerHub). So, if we want to ship an emergency release, we have to circumvent this limitation. A simple way is to trick the script by bumping the date of the /etc/apt/sources.list.d/debian.sources and /etc/apt/sources.list files.

• While we talk about image reproducibility, we can't actually achieve the exact same SHA-256 hash for two different image builds. That's because the file timestamps in the image layers will differ, depending on when the build took place. The rest of the image though (file contents, permissions, manifest) should be byte-for-byte the same. A simple way to check this is with the diffoci tool, and specifically this invocation:

  ./diffoci diff podman://<new_image_tag> podman://<old_image_tag> --ignore-timestamp --verbose