feat: support flat repos #97
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
See previous discussions in #86 |
jjmaestro
force-pushed
the
feat-support-flat-repos
branch
from
bbf5d62 to
88daaa6
Compare
jjmaestro
force-pushed
the
feat-support-flat-repos
branch
from
88daaa6 to
101a90e
Compare
jjmaestro
force-pushed
the
feat-support-flat-repos
branch
4 times, most recently
from
d63d6c2 to
15e865a
Compare
jjmaestro
force-pushed
the
feat-support-flat-repos
branch
2 times, most recently
from
b2bf164 to
bda537d
Compare
45509b0 added `version_constraint` to `apt_deb_repository.bzl` dependencies but it was missing from its `bzl_library` `deps`.
* Remove unnecessary state struct, just pass repository struct to the resolver methods * Remove unused resolve_package from the public API
While working with some flaky mirrors and trying to figure out why they were failing I found the _fetch_package_index code a bit hard to follow so here's my attempt at streamlining it a bit: * Change the general flow of the for-loop so that we can directly set the reasons for failure in failed_attempts * Remove both integrity as an argument and as a return value since neither is ever used. * return the content of the Packages index instead of the path so that (1) we don't need rctx anywhere else and (2) is easier to mock. * Reword failure messages adding more context and debug information * Shorter lines and templated strings, trying to make the code easier to read and follow.
Debian supports uncompressed Package indexes, see: https://wiki.debian.org/DebianRepository/Format#Compression_of_indices Credit to @antholeole (GoogleContainerTools#80) and @wibblenut (GoogleContainerTools#104).
Refactor _version_relop into a compare method in version.bzl plus a VERSION_OPERATORS dict so that (1) we use the operator strings everywhere and (2) we can use the keys to validate the operators.
jjmaestro
force-pushed
the
feat-support-flat-repos
branch
from
bda537d to
c2fd929
Compare
`version_constraint.bzl` is a bunch of utils for package version dependency. Either it has enough entity to stand by itself or the functionality should be folded into `version.bzl` and `package_index.bzl` and/or `package_resolution.bzl`.
Tests should match the file they are testing
Cleanup the version constraint parsing using the VERSION_OPERATORS that we now have in the version struct plus also validating the version being parsed.
* Avoid leaking unresolved dependencies. This is only used to print a warning so it should be self-contained in apt_dep_resolver.bzl * Break up the long lines in comments * Improve _ITERATION_MAX_ fail message * Remove the "# buildifier: disable=print" since it's no longer needed. * rename resolve_all to resolve * reorder (name, version, arch) args to match the order of the index keys (arch, name, version)
* Refactor the nested dict get-set logic into its own nested_dict struct * Remove all the (now) unnecessary wrappers like _package_versions, _virtual_packages, etc. * Fix get() method to return default_value if no keys are given * Add add() to nested dict struct so that we can simplify the virtual package logic in apt_deb_repository.bzl _add_package
* Move Provides validation from _is_satisfied_by ("runtime") to its own
parse_provides method (validate at parse time)
* Encapsulate the 'provided_version' logic here so that the code outside
can be simplified.
* Rename _is_satisfied_by parameters for clarity
Once the validation of 'Provides' version is removed, we can simply use version_lib.compare like we do with the non-virtual packages. Note that the would-be arguments to is_satisfied_by are now inverted so that the version comparison matches the order of the other version comparisons in _resolve_package.
It's clear that the inside of the loop will never return a package if there's no version provided so we can gate the whole for-loop with `if version` and that greatly cleans the whole loop.
* Add testing for apt_deb_repository mocking the external / side effects (downloads, decompression, etc). * Rename _parse_repository back to _parse_package_index * Add test_util.asserts_dict_equals
Cleanup apt_dep_resolver_test removing the apt_deb_repository mock, it's not really needed once we have proper testing and mocking of apt_deb_repository that we can reuse.
* Refactor lockfile into v2 and add tests. The v2 lockfile format:
* uses the nested_dict struct to store the packages so it doesn't need
the fast_package_lookup dict.
* has the dependencies sorted so the lockfile now has stable
serialization and the diffs of the lock are actually usable and
useful to compare with the changes to the manifest.
* removes the package and dependency key from the lockfile and moves
it to an external function in deb_import.bzl (make_deb_import_key).
* Remove add_package_dependency from the lockfile API. Now, the package
dependencies are passed as an argument to add_package. This way, the
lockfile functionality is fully contained in lockfile.bzl and e.g. we
can remove the "consistency checks" that were only needed because
users could forget to add the dependency as a package to the lockfile.
* Ensure backwards-compatibility by internally converting lock v1 to v2.
Also, when a lock is set and it's in v1 format, there's a reminder
that encourages the users to run @repo//:lock to update the lockfile
format.
* Move all of the "package logic" to pkg.bzl
* Add tests for pkg.bzl
* Add mock_value struct to mocks to organize the large mock values used
for testing lockfile, etc.
By separating the migration from the previous commit we get to 1. in the previous commit, run all tests with the new code while locks are still v1 2. update the locks n this commit to V2 so we can then re-run all tests in the final state.
Cleanup deb_resolve.bzl and apt_deb_repository.bzl by moving all of the manifest functionality to a separate manifest.bzl file where we now do all of the work to generate the lock: manifest parsing, validation and the apt repository and package resolution. IMHO this is how it should be because the lock is the "frozen state" of the manifest. * _parse() parses the YAML * _from_dict validates the manifest dict and does the rest of the changes that we need to produce a manifest struct * add extra validations for e.g. duplicated architectures, invalid architectures, etc. Also, all validations are run and failures are printed before actually failing so more of the manifest is validated at the same time. * _lock is the only method that's exposed to the outside and it encapsulates all of the other parts, calling _from_dict and all of the package index and resolution, to produce the lock file. * move get_dupes to util.bzl and add tests * refactor the "source" struct into the new manifest where we can now centralize a lot of the structure and logic spread across multiple parts of the code. * remove yq_toolchain_prefix since it's always "yq" and, looking at GH code search, this seems to be a copy-paste leftover from rules_js (or the other way around)... the code is always the same and it never receives a string different from "yq".
…t.bzl Refactor the package repo templates into their own methods and massively cleanup the `for`-loop in `_deb_package_index_impl`. IMHO overall now there's a much better and clear separation of concerns between the "deb_translate_lock" repo and the "package repos" (`apt/private/deb_import.bzl`).
Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.
Although the Debian repo spec for 'Filename' (see https://wiki.debian.org/DebianRepository/Format#Filename) clearly says that 'Filename' should be relative to the base directory of the repo and should be in canonical form (i.e. without '.' or '..') there are cases where this is not honored. In those cases we try to work around this by assuming 'Filename' is relative to the sources.list directory/ so we combine them and normalize the new 'Filename' path. Note that, so far, only the NVIDIA CUDA repos needed this workaround so maybe this heuristic will break for other repos that don't conform to the Debian repo spec.
jjmaestro
force-pushed
the
feat-support-flat-repos
branch
from
c2fd929 to
ea90090
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Stacked on top of #96
Fixes issue #56 and #66
Follow-up and credit to @alexconrey (PR #55), @ericlchen1 (PR #64) and @benmccown (PR #67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅
Problem
Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax:
"canonical": (see https://wiki.debian.org/DebianRepository/Format#Overview)
flat: (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format)
Per the spec,
Thus, the URL logic in
_fetch_package_index()is incorrect for these repos and it always fails to fetch the Package index.Solution
Just use the Debian sources.list convention in the
sourcessection of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in'/'or a(dist, component, ...)structure the_fetch_package_index ()and other internal logic will know whether the source is a canonical or a flat repo.For example:
Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos.
You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests.
Note
This PR also fixes an issue with NVIDIA CUDA flat repos that don't follow the Debian repo spec and have invalid
'Filename'paths.The Debian repo spec for
'Filename'says:However, there are cases where this is not honored. In those cases we try to work around this by assuming
'Filename'is relative to thesources.listdirectory/so we combine them and normalize the new'Filename'path.Note that, so far, only the NVIDIA CUDA repos needed this workaround so maybe this heuristic will break for other repos that don't conform to the Debian repo spec.