FR: An alternative API design for easier apt.install

[thesayyn]

apt = use_extension("@rules_distroless//apt:extensions.bzl", "apt")
apt.configure(
    architectures = ["amd64", "arm64"]
)
apt.sources_list(
    "deb [arch=amd64,arm64] https://snapshot.ubuntu.com/ubuntu/20240301T030400Z noble main",
    "deb [arch=amd64,arm64] https://snapshot.ubuntu.com/ubuntu/20240301T030400Z noble-security main",
    "deb [arch=amd64,arm64] https://snapshot.ubuntu.com/ubuntu/20240301T030400Z noble-updates main",
)
apt.install(
    "ncurses-base",
    "libncurses6",
    "tzdata",
    "coreutils:arm64",
    "libstdc++6:i386"
)
apt.lock(into = ":lock.json")

[jjmaestro]

This is serendipitous 😄

I didn't see this issue until I got notified when you mentioned it in #56 but last week I was also drafting an almost exact API!

I've been thinking about it since my reviews to those old PRs and the issues I found with the architectures. I 200% love this, the current API conflates the architecture of sources with the architecture of install packages. This new API mirrors much closer to how Debian works, I think it will be awesome to have.

TL;DR

Here's a quick overview of how I was drafting a new API, incorporating some of your

• MODULES.bazel

apt = use_extension("@rules_distroless//apt:extensions.bzl", "apt")

apt.sources(
    name = "bookworm",
    file = "//apt/sources/bookworm.sources",
)

apt.sources(
    name = "docker",
    file = "//apt/sources/docker.list",
)

apt.install(
    name = "base"
    # architectures = ["arm64"],
    sources = ["@apt//sources/bookworm"],
    packages = "//apt/packages/bookworm.yaml",
    lock = "//apt/packages/bookworm.lock.json",
)

# defaults to the "nolock = True" behavior
apt.install(
    name = "docker"
    # architectures = ["arm64"],
    sources = ["@apt//sources/docker"],
    packages = "//apt/packages/docker.yaml",
)

use_repo(apt, "apt")

• Commands:
  These commands can be optional / removed if FR: Detect manifest changes and automatically run lock #130 is implemented.

# update sources repo
$ bazel run @apt//sources/bookworm:update

# make / update install lock
$ bazel run @apt//install/base:lock

• //apt/sources/bookworm.sources (DEB822 .sources format):

Types: deb
Architectures: amd64 arm64
URIs: http://deb.debian.org/debian
Suites: bookworm bookworm-updates
Components: main contrib non-free non-free-firmware

Types: deb
Architectures: amd64 arm64
URIs: http://deb.debian.org/debian-security
Suites: bookworm-security
Components: main contrib non-free non-free-firmware

• //apt/sources/docker.list (one-line .list format):

deb [arch=amd64,arm64] https://download.docker.com/linux/debian bookworm stable

• //apt/packages/bookworm.yaml:

packages:
  - coreutils  # for commands like `ls`
  # for apt list --installed
  - dpkg
  - apt
  - perl
  # docker packages
  - docker

• //apt/packages/docker.yaml:

packages:
  - docker

• BUILD

load("@rules_oci//oci:defs.bzl", "oci_image")

oci_image(
    name = "base",
    architecture = select({
        "@platforms//cpu:arm64": "arm64",
        "@platforms//cpu:x86_64": "amd64",
    }),
    os = "linux",
    tars = [
        "@apt//install/bookworm",
    ],
)

oci_image(
    name = "docker",
    base = ":base",
    tars = [
        "@apt//install/docker",
    ],
)

Comments

Here's a long version of the TL;DR above with more explanations. Also, I've used folded sections to avoid a "wall of text".

apt.sources: support DEB822 .sources format

apt.sources: support DEB822 .sources format

I think the new API should support the "new" ([since 2015! apt v1.1]) DEB822 format.

This format is the same as other Debian files (e.g. the metadata files that apt will download from the configured sources) so it should be easy to support. Also, the old one-line .list format has a few disadvantages, ends up being more verbose for complex setups and/or complicates the parsing so it would be nice to have an alternative.

Following Debian's convention re. sources file extension, I think apt.sources should receive a file and, depending on the extension use the old or new format:

• //apt/sources/bookworm.sources (DEB822)::

Types: deb
Architectures: amd64 arm64
URIs: http://deb.debian.org/debian
Suites: bookworm bookworm-updates
Components: main contrib non-free non-free-firmware

Types: deb
Architectures: amd64 arm64
URIs: http://deb.debian.org/debian-security
Suites: bookworm-security
Components: main contrib non-free non-free-firmware

• //apt/sources/docker.list (one-line):

deb [arch=amd64,arm64] https://download.docker.com/linux/debian bookworm stable

apt.sources(
    name = "bookworm",
    file = "//apt/sources/bookworm.sources",
)

apt.sources(
    name = "docker",
    file = "//apt/sources/docker.list",
)

apt "monorepo"

apt "monorepo"

Could it be possible (and would it make sense) to have everything under one @apt "monorepo" with different packages for sources, install, ...?

Maybe something like:

@apt//
    sources/
        <SOURCE_NAME>/
            <ARCH>/
                Packages.gz
            pool/
                <PACKAGE_NAME>/
                    <ARCH>/
                        <VERSION>/
                            :data
                            :control
    install/
        <INSTALL_NAME>/
            <PACKAGE_NAME>/
                <ARCH>/
                    :data --> //sources/<SOURCE_NAME>/pool/<PACKAGE_NAME>/<ARCH>/<VERSION>:data
                    :control

It would centralize access, removes potential name collisions, simplifies not having to do use_repo for every repo (or having one very long use_repo(...)). Plus I think it would also simplify the access to e.g. the resolve lock and remove the "reported incorrect imports of repositories via use_repo" warning for the _resolve repos in the tests.

And it would centralize and "mirror apt commands" like:

• bazel run @apt//source/<SOURCE_NAME>:update to update the sources, downloading the latest Packages index (and, if we had some internal parsing and generation of an index lock, re-generate it).
• bazel run @apt//install/<INSTALL_NAME>:lock to lock the installation (equivalent to the current bazel run @INSTALL_NAME//:lock).

sources

Similar to how Debian supports multiple .list and .sources files in /etc/apt/sources.list.d, we could have them all under @apt//sources:

use_repo(apt, "apt")

apt.install(
    name = "noble_core",
    sources = [
        "@apt//sources/noble",
        "@apt//sources/foo",
    ],
    ...
)

pool

Similar to how the Debian mirrors organize their structure, they separate the metadata from the pool of files to download. Thus, when resolving a file from a source, it could be mirrored to @apt//sources/<SOURCE_NAME>/pool/<PACKAGE_NAME>.

This would be the equivalent to the current package repo. That is, instead of having @<PACKAGE_NAME>~<VERSION>... repos, they could be namespaced under pool following the same structure as the package repos and the v2 lock that I proposed in #95: <PACKAGE_NAME>/<ARCH>/<VERSION>

install package

With this structure, packages are only downloaded once into the monorepo and all the installs are references to them (like currently, but instead of pointing at per-package repos it would be one @apt repo and Bazel package structure).

apt.install package list

apt.install package list

I think the packages section of the manifest file can still be valuable, it will definitely help un-clutter MODULES.bazel and organize stuff in medium-to-big projects.

I would still use YAML because it supports comments which can be useful when documenting why certain packages or certain constraints are used, e.g.

• //apt/install/packages.yaml:

packages:
# main packages
"coreutils"
  - "dpkg"
  - "apt"
- foo # needed because XYZ
- bar
# extra packages
...

apt.install(
    ...
    packages = "//apt/install/packages.yaml",
    ...
)

apt.install lock file

apt.install lock file

Similarly, I think lock should still be part of install "command". I would probably remove nolock and default to running lockless silently, expecting users to know how to generate a lock with the bazel run command.

I'd have a separate command to update the sources though: bazel run @<INSTALL_NAME>//:lock for installs and bazel run @<SOURCE_NAME>//:update for sources, mirroring apt update.

Also, these commands can be optional / removed if #130 is implemented.

multiple install

multiple install

Support multiple installs, each its own repo:

apt.install(
    name = "base",
    packages = "//apt/install/base.yaml",
)

apt.install(
    name = "docker",
    packages = "//apt/install/docker.yaml",
)

use_repo(apt, "noble_core", "noble_extra")

multiple sources

multiple sources

Mirror /etc/apt/sources.list.d/ having many source files:

apt.sources(
    name = "bookworm",
    file = "//apt/sources/bookworm.sources",
)

apt.sources(
    name = "docker",
    file = "//apt/sources/docker.list",
)

sources repos

sources repos

If "no monorepo", I still think each sources should be its own repository so that we can mix-and-match sources in many installs:

apt.sources(
    name = "noble",
    ...
)
use_repo(apt, "noble")

apt.sources(
    name = "bookworm",
    ...
)
use_repo(apt, "bookworm")

apt.sources(
    "foo",
    ...
)
use_repo(apt, "foo")

apt.install(
    sources = ["@noble", "@foo"],
    ...
)

apt.install(
    sources = ["@bookworm", "@foo"],
    ...
)

architectures per-install

architectures per-install

Instead of (or in addition to) apt.configure, there could be architectures per-install:

apt.sources(
    name = "foo",
    "deb [arch=amd64,arm64] ...",
)

apt.sources(
    name = "bar",
    "deb [arch=arm64,s390x] ...",
)

apt.install(
    name = "install_amd64",
    sources = [
        "@foo",
    ],
    architectures = ["amd64"],
    packages = [
        "pkg1",
        "pkg2",
    ],
)

apt.install(
    name = "install_arm64",
    sources = [
        "@foo",
        "@bar",
    ],
    architectures = ["arm64"],
    packages = [
        "pkg2",
    ],
)

I think this could be useful, regardless of and on top of the apt install :<ARCH> syntax, which would still allow something like:

apt.install(
    name = "install_mixed",
    sources = [
        "@foo",
        "@bar",
    ],
    architectures = ["amd64", "arm64"],
    packages = [
        "pkg1:amd64",
        "pkg2",
    ],
)