Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[+] Initial support for OCI (ghcr) #3

Closed
2 tasks
Azathothas opened this issue Jan 5, 2025 · 1 comment
Closed
2 tasks

[+] Initial support for OCI (ghcr) #3

Azathothas opened this issue Jan 5, 2025 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@Azathothas
Copy link
Member

Azathothas commented Jan 5, 2025

With the new metadata now ready at: https://docs.pkgforge.dev/repositories/bincache/metadata
It's time we introduce initial support for pulling OCI images or just their blob
There exists no better library than oras: https://github.com/oras-project/rust-oci-client
I have already tested their CLI: https://github.com/oras-project/oras & and we use it in production for all our ghcr related ops.

Based on the metadata, we have two fields .ghcr_pkg & .ghcr_blob

  • .ghcr_pkg (Contains the registry_url + tag)

In the metadata:

"ghcr_pkg": "ghcr.io/pkgforge/bincache/curl/stunnel/trurl:8.11.1-x86_64-linux",
  • Oras:
$ ls
total 0

$ oras pull "ghcr.io/pkgforge/bincache/curl/stunnel/trurl:8.11.1-x86_64-linux"
#pulls all artifacts (files)
$ ls
total 7.8M
-rw-rw-r-- 1 runner runner  219 Jan  5 07:45 CHECKSUM
-rw-rw-r-- 1 runner runner  303 Jan  5 07:45 CHECKSUM.sig
-rw-rw-r-- 1 runner runner 7.8M Jan  5 07:45 trurl
-rw-rw-r-- 1 runner runner 2.2K Jan  5 07:45 trurl.json
-rw-rw-r-- 1 runner runner  305 Jan  5 07:45 trurl.json.sig
-rw-rw-r-- 1 runner runner  21K Jan  5 07:45 trurl.log
-rw-rw-r-- 1 runner runner  304 Jan  5 07:45 trurl.log.sig
-rw-rw-r-- 1 runner runner  300 Jan  5 07:45 trurl.sig
-rw-rw-r-- 1 runner runner 7.9K Jan  5 07:45 trurl.svg
-rw-rw-r-- 1 runner runner    6 Jan  5 07:45 trurl.version
-rw-rw-r-- 1 runner runner  314 Jan  5 07:45 trurl.version.sig
  • Proposal: Introduce --ghcr (also detect if input starts with ghcr.io) , allow --outdir (Use CWD by default)
  • .ghcr_blob (Contains the registry_url + tag + blobdigest)
  • Difference: .ghcr_pkg pulls ALL artifacts (files) wheras .ghcr_blob is only for a single artifact (file)

In the metadata:

"ghcr_blob": "ghcr.io/pkgforge/bincache/curl/stunnel/trurl@sha256:ff70f40b3add6dcccb91de8b42f6678ae3ff8f38fc046da8e42337e5d0501bc5",
  • Oras:
$ ls
total 0

$ oras blob  fetch "ghcr.io/pkgforge/bincache/curl/stunnel/trurl@sha256:ff70f40b3add6dcccb91de8b42f6678ae3ff8f38fc046da8e42337e5d0501bc5" --output "trurl"
#pulls only a single artifact (file)
#oras enforce we MUST supply an --output file as the blob contains no info on filename, maybe we can just use the value before `@`
#or enforce the same requirement?
$ $ ls
total 7.8M
-rw-rw-r-- 1 runner runner 7.8M Jan  5 07:53 trurl

$ file curl
trurl: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
  • Proposal: Introduce --ghcr (also detect if input starts with ghcr.io) , allow --output (Use CWD by default) [Detect pkg or blob based on if input contains @sha256]

  • Authentication & Headers

GitHub container registry don't allow for anonymous access, but they don't strictly enforce it either, so we can just use arbitrary Authorization: ${HEADER}
Homebrew & our api uses: https://github.com/pkgforge-dev/reverse-proxies/blob/main/api.ghcr.pkgforge.dev/worker.js#L154

"Authorization: Bearer QQ=="

There are also some other headers that need to be passed, but maybe the library does it automatically?
If not, set: https://github.com/pkgforge-dev/reverse-proxies/blob/main/api.ghcr.pkgforge.dev/worker.js#L155

"Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
@QaidVoid QaidVoid added the enhancement New feature or request label Jan 11, 2025
@QaidVoid
Copy link
Member

Added in baf4ea7, 45de669.

The path before the digest is used as default filename for the blob download. Output path can be either the filename or a directory name for blob downloads, but any output path is treated as a directory for package downloads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants