fix: platforms, architectures and dpkg_status after #115 (#120)

* chore: pre-commit run --all-files Ditto * fix: different _PACKAGES, :packages and :dpkg_status per architecture There are cases where the packages differ between architectures. E.g. currently in the `examples/ubuntu_snapshot`, `coreutils` depends on `libssl3` on `amd64` but there's no such dependency on `arm64`. Thus, the list of packages and `dpkg_status` has to be different. This failure wasn't caught because this corner case was only happening in the Ubuntu example which was still using "the old `dpkg_status`" that as done by hand, just passing a shorter list of packages and not the full list of installed packages (as implemented in #115). If the test is migrated, when running without the fix it fails with: ERROR: no such package '@@_main~apt~noble//libssl3/arm64': BUILD file not found in directory 'libssl3/arm64' of external repository @@_main~apt~noble. Add a BUILD file to a directory to mark it as a package. ERROR: /home/nonroot/.cache/bazel/_bazel_nonroot/a08c2e4811c846650b733c6fc815a920/external/_main~apt~noble/libssl3/BUILD.bazel:1:6: no such package '@@_main~apt~noble//libssl3/arm64': BUILD file not found in directory 'libssl3/arm64' of external repository @@_main~apt~noble. Add a BUILD file to a directory to mark it as a package. and referenced by '@@_main~apt~noble//libssl3:libssl3' ERROR: Analysis of target '//examples/ubuntu_snapshot:_noble_index_json' failed; build aborted: Analysis failed I've now moved `examples/ubuntu_snapshot` to use the "new" `dpkg_status`. I've also added an explicit test to check that libssl3 is installed in amd64 and another test to check that it's NOT installed in arm64. * fix: sync examples/debian_snapshot and e2e/smoke test These two have been the same but there's been changes to both that weren't replicated / synced into the other. I've now synced and made a comment to remark that it's important to keep these in-sync because testing in e2e with at least the same base test helps in catching bugs like the previous commit, where platforms repo should have been marked as non-dev. Finally, I've moved both tests to Cloudflare's Debian snapshot. Looks like the change happened in f994712 without any explanation and, as usual, I keep finding Debian snapshot extremely flaky and unreliable. Cloudflare has had some issues in the past (e.g. lagging behind replication for a long time) but these are quite rare and resolve much quicker than the Debian snapshot. * fix: debian_snapshot architecture Regardless of the `platform_transition_filegroup`, the architecture needs a `select` to properly set the architecture in the manifest. I verified this by running the test and inspecting the generated config JSON. Ideally, this should be encoded in a test but I'm not sure how to go about it. * feat: add more Debian architectures `_ARCHITECTURE_MAP[arch] or arch` fails when a Debian arch is not in the map. Thus, add all of the Debian architectures that map to platforms CPU architectures. Add "all" arch as well (plus an additional resolution test). Also: * change Debian's ppc64el mapping to the exact matching platform CPU (ppc64le) * move architecture doc links next to _ARCHITECTURE_MAP and add a bunch more links to the Debian wikis. * fix: starlark_codegen_utils.to_dict_list_attr The list inside the dict should be formatted and indented. * fix: buildifier
GoogleContainerTools · Dec 11, 2024 · 936081d · 936081d
1 parent f994712
commit 936081d
Show file tree

Hide file tree

Showing 20 changed files with 4,876 additions and 1,527 deletions.
diff --git a/MODULE.bazel b/MODULE.bazel
@@ -68,7 +68,7 @@ apt.install(
 )
 apt.install(
     name = "bullseye_nolock",
-    manifest = "//examples/debian_snapshot:bullseye_nolock.yaml",
+    manifest = "//examples/debian_snapshot:bullseye.yaml",
     nolock = True,
 )
 apt.install(
@@ -81,4 +81,9 @@ apt.install(
     manifest = "apt/tests/resolution/security.yaml",
     nolock = True,
 )
-use_repo(apt, "bullseye", "bullseye_nolock", "noble", "resolution_test", "resolution_test_resolve")
+apt.install(
+    name = "arch_all_test",
+    manifest = "apt/tests/resolution/arch_all.yaml",
+    nolock = True,
+)
+use_repo(apt, "arch_all_test", "arch_all_test_resolve", "bullseye", "bullseye_nolock", "noble", "resolution_test", "resolution_test_resolve")
diff --git a/README.md b/README.md
@@ -4,20 +4,17 @@ Bazel helper rules to aid with some of the steps needed to create a Linux /
 Debian installation. These rules are designed to replace commands such as
 `apt-get install`, `passwd`, `groupadd`, `useradd`, `update-ca-certificates`.
 
-> [!CAUTION]
-> `rules_distroless` is currently in beta and does not yet offer a stable
+> [!CAUTION] > `rules_distroless` is currently in beta and does not yet offer a stable
 > Public API. However, many users are already successfully using it in
 > production environments. Check [Adopters](#adopters) to see who's already
 > using it.
 
-
 # Usage
 
 ## Bzlmod (Bazel 6+)
 
 > [!NOTE]
-> If you are using Bazel 6 you need to enable Bzlmod by adding `common
-> --enable_bzlmod` to `.bazelrc`
+> If you are using Bazel 6 you need to enable Bzlmod by adding `common --enable_bzlmod` to `.bazelrc`
 > If you are using Bazel 7+ [it's enabled by default].
 
 Add the following to your `MODULE.bazel` file:
@@ -34,6 +31,7 @@ provides to override repos.
 
 You can use [`git_override`], [`archive_override`], etc (or
 [`local_path_override`] if you want to test a local patch):
+
 ```starlark
 bazel_dep(name = "rules_distroless", version = "0.3.9")
 
@@ -100,8 +98,8 @@ can use it to get rid of the warning.
 > [!CAUTION]
 > GitHub source archives don't have a strong guarantee on the sha256 stability.
 > Check Github's [Update on the future stability of source code archives and
-> hashes] for more information.
-
+>
+> > hashes] for more information.
 
 # Examples
 
@@ -124,7 +122,6 @@ We also have `distroless`-specific rules that could be useful:
 - [dpkg_statusd](/examples/statusd): creates a `/var/lib/dpkg/status.d`
   package database for scanners to discover installed packages.
 
-
 # Public API Docs
 
 To read more specific documentation for each of the rules in the repo please
@@ -136,8 +133,8 @@ check the following docs:
 - [rules](/docs/rules.md): various helper rules to aid with creating a Linux /
   Debian installation from scratch.
 
-
 # Contributing
+
 This ruleset is primarily funded to support [Google's `distroless` container
 images]. We may not work on feature requests that do not support this mission.
 
@@ -146,23 +143,22 @@ align with the project goals (e.g. add support for a different compression
 format) and may reject requests that do not (e.g. supporting other packaging
 formats other than `.deb`).
 
-
 # Adopters
+
 - [Google's `distroless` container images]
 - [Arize AI](https://www.arize.com)
 
 > [!TIP]
 > Are you using `rules_distroless`? Please send us a Pull Request to add your
 > project or company name here!
 
-
 [it's enabled by default]: https://blog.bazel.build/2023/12/11/bazel-7-release.html#bzlmod
-[Bazel Central Registry]: https://registry.bazel.build/modules/rules_distroless
+[bazel central registry]: https://registry.bazel.build/modules/rules_distroless
 [`git_override`]: https://bazel.build/versions/6.0.0/rules/lib/globals#git_override
 [`archive_override`]: https://bazel.build/versions/6.0.0/rules/lib/globals#archive_override
 [`local_path_override`]: https://bazel.build/versions/6.0.0/rules/lib/globals#local_path_override
-[Bzlmod migration guide]: https://bazel.build/external/migration
-[`rules_distroless` Github releases page]: https://github.com/GoogleContainerTools/rules_distroless/releases
-[Update on the future stability of source code archives and hashes]: https://github.blog/2023-02-21-update-on-the-future-stability-of-source-code-archives-and-hashes
-[Google's `distroless` container images]: https://github.com/GoogleContainerTools/distroless
-[Arize AI]: https://www.arize.com
+[bzlmod migration guide]: https://bazel.build/external/migration
+[`rules_distroless` github releases page]: https://github.com/GoogleContainerTools/rules_distroless/releases
+[update on the future stability of source code archives and hashes]: https://github.blog/2023-02-21-update-on-the-future-stability-of-source-code-archives-and-hashes
+[google's `distroless` container images]: https://github.com/GoogleContainerTools/distroless
+[arize ai]: https://www.arize.com
diff --git a/apt/private/deb_translate_lock.bzl b/apt/private/deb_translate_lock.bzl
@@ -1,6 +1,5 @@
 "repository rule for generating a dependency graph from a lockfile."
 
-load("@bazel_skylib//lib:new_sets.bzl", "sets")
 load(":lockfile.bzl", "lockfile")
 load(":starlark_codegen_utils.bzl", "starlark_codegen_utils")
 load(":util.bzl", "util")
@@ -45,22 +44,34 @@ load("@rules_distroless//distroless:defs.bzl", "flatten")
 
 exports_files(['packages.bzl'])
 
+# Map Debian architectures to platform CPUs.
+#
+# For more info on Debian architectures, see:
+#     * https://wiki.debian.org/SupportedArchitectures
+#     * https://wiki.debian.org/ArchitectureSpecificsMemo
+#     * https://www.debian.org/releases/stable/amd64/ch02s01.en.html#idm186
+#
+# For more info on Bazel's platforms CPUs see:
+#     * https://github.com/bazelbuild/platforms/blob/main/cpu/BUILD
 _ARCHITECTURE_MAP = {{
     "amd64": "x86_64",
-    "arm64": "aarch64",
-    "ppc64el": "ppc",
-    "mips64el": "mips64"
+    "arm64": "arm64",
+    "ppc64el": "ppc64le",
+    "mips64el": "mips64",
+    "s390x": "s390x",
+    "i386": "x86_32",
+    "armhf": "armv7e-mf",
+    "all": "all",
 }}
 
 _ARCHITECTURES = {architectures}
 
-# See officially supported platforms at https://www.debian.org/releases/stable/armel/ch02s01.en.html
 [
    config_setting(
     name = os + "_" + arch,
     constraint_values = [
        "@platforms//os:" + os,
-       "@platforms//cpu:" + (_ARCHITECTURE_MAP[arch] or arch),
+       "@platforms//cpu:" + _ARCHITECTURE_MAP[arch],
     ],
   )
   for os in ["linux"]
@@ -80,19 +91,19 @@ _PACKAGES = {packages}
 # Creates /var/lib/dpkg/status with installed package information.
 dpkg_status(
     name = "dpkg_status",
-    controls = [
-        "//%s:control" % package
-        for package in _PACKAGES
-    ],
+    controls = select({{
+        "//:linux_%s" % arch: ["//%s:control" % package for package in packages]
+        for arch, packages in _PACKAGES.items()
+    }}),
     visibility = ["//visibility:public"],
 )
 
 filegroup(
     name = "packages",
-    srcs = [
-        "//%s" % package
-        for package in _PACKAGES
-    ],
+    srcs = select({{
+        "//:linux_%s" % arch: ["//%s" % package for package in packages]
+        for arch, packages in _PACKAGES.items()
+    }}),
     visibility = ["//visibility:public"],
 )
 
@@ -131,8 +142,8 @@ def _deb_translate_lock_impl(rctx):
             package_defs.append("   pass")
 
     # TODO: rework lockfile to include architecure information
-    architectures = sets.make()
-    packages = sets.make()
+    architectures = {}
+    packages = {}
 
     for (package) in lockf.packages():
         package_key = lockfile.make_package_key(
@@ -141,8 +152,13 @@ def _deb_translate_lock_impl(rctx):
             package["arch"],
         )
 
-        sets.insert(architectures, package["arch"])
-        sets.insert(packages, package["name"])
+        if package["arch"] not in architectures:
+            architectures[package["arch"]] = []
+        architectures[package["arch"]].append(package["name"])
+
+        if package["name"] not in packages:
+            packages[package["name"]] = []
+        packages[package["name"]].append(package["arch"])
 
         if not lock_content:
             package_defs.append(
@@ -174,27 +190,27 @@ def _deb_translate_lock_impl(rctx):
         )
 
     # TODO: rework lockfile to include architecure information and merge these two loops
-    for (package) in lockf.packages():
+    for package_name, package_archs in packages.items():
         rctx.file(
-            "%s/BUILD.bazel" % (package["name"]),
+            "%s/BUILD.bazel" % (package_name),
             _PACKAGE_TEMPLATE.format(
-                target_name = package["name"],
+                target_name = package_name,
                 data_targets = starlark_codegen_utils.to_dict_attr({
-                    "//:linux_%s" % arch: "//%s/%s" % (package["name"], arch)
-                    for arch in architectures._values
+                    "//:linux_%s" % arch: "//%s/%s" % (package_name, arch)
+                    for arch in package_archs
                 }),
                 control_targets = starlark_codegen_utils.to_dict_attr({
-                    "//:linux_%s" % arch: "//%s/%s:control" % (package["name"], arch)
-                    for arch in architectures._values
+                    "//:linux_%s" % arch: "//%s/%s:control" % (package_name, arch)
+                    for arch in package_archs
                 }),
             ),
         )
 
     rctx.file("packages.bzl", "\n".join(package_defs))
     rctx.file("BUILD.bazel", _ROOT_BUILD_TMPL.format(
         target_name = util.get_repo_name(rctx.attr.name),
-        packages = starlark_codegen_utils.to_list_attr(packages._values),
-        architectures = starlark_codegen_utils.to_list_attr(architectures._values),
+        packages = starlark_codegen_utils.to_dict_list_attr(architectures),
+        architectures = starlark_codegen_utils.to_list_attr(architectures.keys()),
     ))
 
 deb_translate_lock = repository_rule(

diff --git a/apt/private/starlark_codegen_utils.bzl b/apt/private/starlark_codegen_utils.bzl
@@ -34,6 +34,7 @@ def _to_dict_list_attr(dict, indent_count = 0, indent_size = 4, quote_key = True
     result = "{"
     for k, v in dict.items():
         key = "\"{}\"".format(k) if quote_key else k
+        v = _to_list_attr(v, indent_count + 1, indent_size)
         result += "\n%s%s%s: %s," % (indent, tab, key, v)
     result += "\n%s}" % indent
     return result

diff --git a/apt/tests/resolution/BUILD.bazel b/apt/tests/resolution/BUILD.bazel
@@ -15,3 +15,33 @@ assert_contains(
     actual = ":pick_libuuid_version",
     expected = "2.38.1-5+deb12u1",
 )
+
+jq(
+    name = "pick_quake_arch",
+    srcs = [
+        "@arch_all_test_resolve//:lockfile",
+    ],
+    args = ["-rj"],
+    filter = '.packages | map(select(.name == "quake")) | .[0].arch',
+)
+
+assert_contains(
+    name = "test_quake_arch",
+    actual = ":pick_quake_arch",
+    expected = "all",
+)
+
+jq(
+    name = "pick_quake_version",
+    srcs = [
+        "@arch_all_test_resolve//:lockfile",
+    ],
+    args = ["-rj"],
+    filter = '.packages | map(select(.name == "quake")) | .[0].version',
+)
+
+assert_contains(
+    name = "test_quake_version",
+    actual = ":pick_quake_version",
+    expected = "73",
+)
diff --git a/apt/tests/resolution/arch_all.yaml b/apt/tests/resolution/arch_all.yaml
@@ -0,0 +1,11 @@
+version: 1
+
+sources:
+  - channel: bookworm main contrib
+    url: https://snapshot-cloudflare.debian.org/archive/debian/20240401T030239Z
+
+archs:
+  - all
+
+packages:
+  - quake