Package not found when using a base image with an extra repository

[rodrigol-chan]

mkosi commit the issue has been seen with

40bf1fb

Used host distribution

Ubuntu 24.04

Used target distribution

Ubuntu 22.04

Linux kernel version used

6.8.0-39-generic

CPU architectures issue was seen on

None

Unexpected behaviour you saw

I expect consul to be installable. When using an image with an extra repository as a base image, I expect packages from that repository to be installable. It seems that /etc is mounted from some pkgmngr directory that doesn't include the changes I did to the base image, and so the repository is never known in the derived image.

hashicorp.list and hashicorp-archive-keyring.gpg are mostly as described in https://www.hashicorp.com/official-packaging-guide

Used mkosi config

# Directory structure:
# ./mkosi.images
# ./mkosi.images/base
# ./mkosi.images/base/mkosi.skeleton
# ./mkosi.images/base/mkosi.skeleton/etc
# ./mkosi.images/base/mkosi.skeleton/etc/apt
# ./mkosi.images/base/mkosi.skeleton/etc/apt/sources.list.d
# ./mkosi.images/base/mkosi.skeleton/etc/apt/sources.list.d/hashicorp.list
# ./mkosi.images/base/mkosi.skeleton/etc/apt/trusted.gpg.d
# ./mkosi.images/base/mkosi.skeleton/etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg
# ./mkosi.images/base/mkosi.conf
# ./mkosi.images/consul-server
# ./mkosi.images/consul-server/mkosi.conf
# ./mkosi.conf

# ./mkosi.conf
[Output]
Format=none
[Distribution]
Distribution=ubuntu
Release=jammy
Architecture=x86-64

# ./mkosi.images/base/mkosi.conf
[Output]
Format=tar
[Content]
Packages=linux-image-generic,grub-efi-amd64-signed,shim-signed

# ./mkosi.images/consul-server/mkosi.conf
[Config]
Dependencies=base
[Output]
Format=disk
[Content]
BaseTrees=%O/base
Bootable=true
Bootloader=grub
UnifiedKernelImages=no
Packages=consul

mkosi output

https://gist.github.com/rodrigol-chan/c1243c4eb7a4823b241c18d0250133a7

[DaanDeMeyer]

This is working as expected, we don't use any package manager configuration from base trees by default. You can pick it up explicitly by using PackageManagerTrees=%O/base/etc/apt:/etc/apt in the consul-server image.

[rodrigol-chan]

Thank you for the quick reply! It now fails with the following two errors. Full mkosi --debug.

...
‣   Creating cpio archive /home/rodrigo/.cache/mkosi/mkosi-workspace-f_k0p8k8/staging/initrd.cpio…
‣ + cpio --create --reproducible --null --format=newc --quiet --directory /home/rodrigo/.cache/mkosi/mkosi-workspace-f_k0p8k8/root
‣   Compressing /home/rodrigo/.cache/mkosi/mkosi-workspace-f_k0p8k8/staging/initrd.cpio with zstd
‣ + zstd -q -3 -T0 --stdout -
‣ + stat --file-system --format %T /home/rodrigo/.cache/mkosi/mkosi-workspace-f_k0p8k8/root
‣ + btrfs subvolume delete /home/rodrigo/.cache/mkosi/mkosi-workspace-f_k0p8k8/root
Delete subvolume 575 (no-commit): '/home/rodrigo/.cache/mkosi/mkosi-workspace-f_k0p8k8/root'
ERROR: Could not destroy subvolume/snapshot: Operation not permitted
WARNING: deletion failed with EPERM, you don't have permissions or send may be in progress
‣ + rm -rf -- /home/rodrigo/.cache/mkosi/mkosi-workspace-f_k0p8k8/root
‣   /home/rodrigo/.cache/mkosi/mkosi-workspace-oxbd3d4z/initrd.cpio.zst size is 32.8M, consumes 32.8M.
‣ + rm -rf -- /home/rodrigo/.cache/mkosi/mkosi-workspace-f_k0p8k8
...

‣  Fixing ownership of package manager cache directory
‣ + chown --recursive 65436:65436 '/home/rodrigo/.cache/mkosi/ubuntu~jammy~x86-64/cache/apt'
‣ + chown --recursive 65436:65436 '/home/rodrigo/.cache/mkosi/ubuntu~jammy~x86-64/lib/apt'
‣ + tput cnorm
‣ + tput smam
Traceback (most recent call last):
  File "/nix/store/s498zn7yyqa53f83462zw67wbdg6204n-mkosi-24.3/lib/python3.12/site-packages/mkosi/run.py", line 60, in uncaught_exception_handler
    yield
  File "/nix/store/l014xp1qxdl6gim3zc0jv3mpxhbp346s-python3-3.12.4/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/s498zn7yyqa53f83462zw67wbdg6204n-mkosi-24.3/lib/python3.12/site-packages/mkosi/__main__.py", line 39, in main
    run_verb(args, images, resources=resources)
  File "/nix/store/s498zn7yyqa53f83462zw67wbdg6204n-mkosi-24.3/lib/python3.12/site-packages/mkosi/__init__.py", line 4936, in run_verb
    check_inputs(config)
  File "/nix/store/s498zn7yyqa53f83462zw67wbdg6204n-mkosi-24.3/lib/python3.12/site-packages/mkosi/__init__.py", line 2866, in check_inputs
    if not tree.source.exists():
           ^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/l014xp1qxdl6gim3zc0jv3mpxhbp346s-python3-3.12.4/lib/python3.12/pathlib.py", line 860, in exists
    self.stat(follow_symlinks=follow_symlinks)
  File "/nix/store/l014xp1qxdl6gim3zc0jv3mpxhbp346s-python3-3.12.4/lib/python3.12/pathlib.py", line 840, in stat
    return os.stat(self, follow_symlinks=follow_symlinks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/home/rodrigo/tmp/sandbox/mkosi.output/base/etc/apt'

I've amended the configuration as you suggested. I had to set Format=directory on the base image. For reference, the full configuration is:

────────────────────────────────────────────────────────────────────────────────
File: mkosi.conf
────────────────────────────────────────────────────────────────────────────────
[Output]
Format=none

[Distribution]
Distribution=ubuntu
Release=jammy
Architecture=x86-64
────────────────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────
File: mkosi.images/base/mkosi.conf
────────────────────────────────────────────────────────────────────────────────
[Output]
Format=directory

[Content]
Packages=linux-image-generic,grub-efi-amd64-signed,shim-signed
────────────────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────
File: mkosi.images/consul-server/mkosi.conf
────────────────────────────────────────────────────────────────────────────────
[Config]
Dependencies=base

[Distribution]
PackageManagerTrees=%O/base/etc/apt:/etc/apt

[Output]
Format=disk

[Content]
BaseTrees=%O/base
Bootable=true
Bootloader=grub
UnifiedKernelImages=no
Packages=consul
────────────────────────────────────────────────────────────────────────────────

[DaanDeMeyer]

@rodrigol-chan Please check the access mode on mkosi.skeleton/etc/apt. It's probably too restrictive. It should be 755 for this to work.

[rodrigol-chan]

Ah, that was it. Thank you very much!