cobbler sync fails with containerized server

[ppanon2022]

After running configure-tftpsync.sh to set up proxies in the cobbler configuration, running cobbler sync fails to replicate the tftp data to the proxies.

While it says the task completes

# cobbler sync
task started: 2024-11-01_113752_sync
task started (id=Sync, time=Fri Nov  1 07:37:52 2024)
syncing all
Waiting sync_lock to be available to perform the sync action (this might take some time)
running pre-sync triggers
cleaning trees
removing: /srv/www/cobbler/images/Rocky8:1:SemtechCorporation
Successfully removed "/srv/tftpboot/pxelinux.cfg/default"
removing: /srv/tftpboot/grub/images
removing: /srv/tftpboot/grub/i386-pc
removing: /srv/tftpboot/grub/x86_64-efi
Successfully removed "/srv/tftpboot/grub/bootia32.efi"
Successfully removed "/srv/tftpboot/grub/grub.0"
Successfully removed "/srv/tftpboot/grub/grubx86.efi"
Successfully removed "/srv/tftpboot/grub/shim.efi"
removing: /srv/tftpboot/grub/system
removing: /srv/tftpboot/grub/system_link
Successfully removed "/srv/tftpboot/grub/grub.cfg"
Successfully removed "/srv/tftpboot/grub/grub.cfg.rpmsave"
Successfully removed "/srv/tftpboot/grub/local_efi.cfg"
Successfully removed "/srv/tftpboot/grub/local_efi.cfg.rpmsave"
Successfully removed "/srv/tftpboot/grub/local_legacy.cfg"
Successfully removed "/srv/tftpboot/grub/local_legacy.cfg.rpmsave"
Successfully removed "/srv/tftpboot/grub/local_powerpc-ieee1275.cfg"
Successfully removed "/srv/tftpboot/grub/local_powerpc-ieee1275.cfg.rpmsave"
Successfully removed "/srv/tftpboot/grub/i386_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/x86_64_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/ia64_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/ppc_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/ppc64_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/ppc64le_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/ppc64el_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/s390_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/s390x_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/arm_menu_items.cfg"
Successfully removed "/srv/tftpboot/grub/aarch64_menu_items.cfg"
removing: /srv/tftpboot/images/Rocky8:1:SemtechCorporation
copying bootloaders
running: ['rsync', '-rpt', '--copy-links', '--exclude=.cobbler_postun_cleanup', '/var/lib/cobbler/loaders/', '/srv/tftpboot']
received on stdout:
running: ['rsync', '-rpt', '--copy-links', '--exclude=README.grubconfig', '/var/lib/cobbler/grub_config/', '/srv/tftpboot']
received on stdout:
copying distros to tftpboot
copying files for distro: Rocky8:1:SemtechCorporation
trying hardlink /srv/www/distributions/1-Rocky8/images/pxeboot/vmlinuz -> /srv/tftpboot/images/Rocky8:1:SemtechCorporation/vmlinuz
running: /usr/bin/sha1sum /srv/www/distributions/1-Rocky8/images/pxeboot/vmlinuz
received on stdout: 6977f28e6809fa168127cca0d79f0a73c1cbaae4  /srv/www/distributions/1-Rocky8/images/pxeboot/vmlinuz
trying hardlink /srv/www/distributions/1-Rocky8/images/pxeboot/initrd.img -> /srv/tftpboot/images/Rocky8:1:SemtechCorporation/initrd.img
running: /usr/bin/sha1sum /srv/www/distributions/1-Rocky8/images/pxeboot/initrd.img
received on stdout: d98d92d2b49f418a190f9a7ec0061a7d5b1d0403  /srv/www/distributions/1-Rocky8/images/pxeboot/initrd.img
copying images
generating PXE configuration files - this can take a while (to see the progress check the cobbler logs)
generating PXE menu structure
https://cobbler.github.io\nTIMEOUT 200\nTOTALTIMEOUT 6000\nONTIMEOUT local\n\nLABEL local\n        MENU LABEL (local)\n        MENU DEFAULT\n        LOCALBOOT -1\n\nMENU end\n', 'grub': {}}'
copying files for distro: Rocky8:1:SemtechCorporation
trying hardlink /srv/www/distributions/1-Rocky8/images/pxeboot/vmlinuz -> /srv/www/cobbler/images/Rocky8:1:SemtechCorporation/vmlinuz
trying hardlink /srv/www/distributions/1-Rocky8/images/pxeboot/initrd.img -> /srv/www/cobbler/images/Rocky8:1:SemtechCorporation/initrd.img
Writing template files for Rocky8:1:SemtechCorporation
processing boot_files for distro: Rocky8:1:SemtechCorporation
cleaning link caches
running: ['find', '/srv/tftpboot/images/.link_cache', '-maxdepth', '1', '-type', 'f', '-links', '1', '-exec', 'rm', '-f', '{}', ';']
received on stdout:
running post-sync triggers
sync_post_tftp_proxies started - this can take a while (to see the progress check the cobbler logs)
*** TASK COMPLETE ***

the tftpd data isn't visible in the proxies (using podman exec to look at the tftp directory in the uyuni-proxy-tftpd container) and the cobbler log is full of error messages like
[ThreadPoolExecutor-3_12] 2024-11-01T11:38:12 - INFO | Push failed
[ThreadPoolExecutor-3_39] 2024-11-01T11:38:12 - INFO | Push failed
[Thread-4998] 2024-11-01T11:38:12 - ERROR | uploading to proxy myuyproxy1.mydomain failed: HTTP Error 403: Forbidden
[Thread-4963] 2024-11-01T11:38:12 - ERROR | uploading to proxy myuyproxy2.mydomain failed: HTTP Error 403: Forbidden

[aaannz]

Thanks for the report.
Containerized proxies no longer use tftpsync. Instead all data are cached using squid and downloaded on demand.

[ppanon2022]

How does PXE booting for autoinstallable distributions work with squid? Doesn't PXE booting pretty much require TFTP for the initial bootstrap download? And, if that's the case, why is there even an uyuni-proxy-tftpd container in the pod? When was dropping this functionality announced in either community hours or release notes? I remember the deprecation and removal of legacy/traditional (aka jabber-mediated) client connections being mentioned, but not this.

Is there a way to reenable this through some changes to the Apache httpd configuration (since cobbler seems to try to use that pathway) or was the ability of cobbler to update the tftpd service data directory considered too much of a security vulnerability? Or is the issue that apache httpd and tftpd are in separate containers so that httpd can't access the tftpd data directories? Come to think that's last is almost certainly the problem, but shouldn't it be possible for the /var/lib/containers/storage/volumes/uyuni-proxy-tftpboot/ volume to be mapped into both containers? Looks like the z option is what should be used for sharing a volume - is that not working for some reason?

Anyways, as you can tell, I have many questions. :-)

OK, to answer my own questions, you've switched from in.tftpd to the python3-fbtftp package which presumably you've extended to tap into the squid proxy. I can't figure out how it should be working or logging access though, or how to configure it to increase logging. When I try to use a pxeclient, it seems to get the right parameters from the DHCP server, but it's never successful on obtaining the PXE bootstrap file via TFTP. I don't see anything in a packet capture on the proxy server either.

The dhcp server config is the one generated by cobbler

#
# Uyuni PXE config
#
# listening on VLAN 38 NIC due to 10.23.28.0 subnet section
#

#ddns-update-style interim;

allow booting;
allow bootp;

ignore client-updates;
set vendorclass = option vendor-class-identifier;

option system-arch code 93 = unsigned integer 16;

option domain-name "my.domain";
default-lease-time 21600;
max-lease-time 43200;
#
ddns-updates off;
#option TCode code 101 = text;
option TCode "America/Vancouver";
#
# Cobbler managed dhcpd.conf file
# generated from cobbler dhcp.conf template (Thu Oct 31 00:05:50 2024)
# Do NOT make changes to /etc/dhcpd.conf. Instead, make your changes
# in /etc/cobbler/dhcp.template, as /etc/dhcpd.conf will be
# overwritten.
# ******************************************************************

ddns-update-style interim;

allow booting;
allow bootp;

ignore client-updates;
set vendorclass = option vendor-class-identifier;

option system-arch code 93 = unsigned integer 16;

subnet XX.YY.ZZ.0 netmask 255.255.255.0 {
     option routers             XX.YY.ZZ.1;
     option domain-name-servers XX.YY.MM.NNN, XX.YY.MM.NNO;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        XX.YY.ZZ.20 XX.YY.ZZ.200;
     option broadcast-address   XX.YY.ZZ.255;
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                XX.YY.ZZ.15;
     class "pxeclients" {
          match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";

          # Legacy
          if option system-arch = 00:00 {
              filename "grub/grub.0";
          }
          # UEFI-32-1
          if option system-arch = 00:06 {
               # Not supported, no 32 bit UEFI grub executable
              filename "unsupported";
          }
          # UEFI-32-2
          if option system-arch = 00:02 {
              # Not supported, no 32 bit UEFI grub executable
              filename "unsupported";
          }
          # UEFI-64-1
          else if option system-arch = 00:07 {
              filename "grub/grubx64.efi";
          }
          # UEFI-64-2
          else if option system-arch = 00:08 {
              filename "grub/grubx64.efi";
          }
          # UEFI-64-3
          else if option system-arch = 00:09 {
              filename "grub/grubx64.efi";
          }
          # armv7   (aka arm 32 bit)
          else if option system-arch = 00:0a {
              filename "grub/armv7.efi";
          }
          # aarch64 (aka arm 64 bit)
          else if option system-arch = 00:0b {
              filename "grub/grubaa64.efi";
          }
          # RiskV 32 bit
          else if option system-arch = 00:25 {
              #ToDo petitboot loader
              # petitboot should support pxelinux config files
              filename "unsupported";
          }
          #RiskV 32 bit
          else if option system-arch = 00:27 {
              #ToDo petitboot loader
              # petitboot should support pxelinux config files
              filename "unsupported";
          }
          else if option system-arch = 00:0e {
              # FIXME add petitboot support for ppc64(le)
              filename "grub/grub.ppc64le";
          }
          else
          {
              filename "grub/grub.0";
          }
     }
}

The client shows

>>Start PXE over IPv4.
  Station IP address is XX.YY.ZZ.20
  Server IP address is XX.YY.ZZ.15
  NBP filename is grub/grubx64.efi
  NBP filesize is 0 Bytes
  PXE-E99: Unexpected network error:
BdsDxe: failed to load Boot0002 "UEFI PXE v4 (MAC:....)" from PciRoot(...)/Pci(...)/Pci(...)/MAC(....0x1)/IPv4(0.0.0.0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0): Not Found

and then moves on to try to get the EFI bootstrap file over IPv6 and HTTP, which similarly fail.
Thus, it appears to get the proper BOOTP parameters/options needed to fetch the botstrap file, even though I'm not seeing any obvious UDP port 69 packet hits on the proxy.

[aaannz]

How does PXE booting for autoinstallable distributions work with squid? Doesn't PXE booting pretty much require TFTP for the initial bootstrap download?

As you already found out, tftp container is using fbtftp which translates tftp requests to http and reuses existing proxy caching mechanism to cache pxe/uefi pxe data.

And, if that's the case, why is there even an uyuni-proxy-tftpd container in the pod?

With proxy we split all services to their separate containers. So tftp service has it's own container.

When was dropping this functionality announced in either community hours or release notes? I remember the deprecation and removal of legacy/traditional (aka jabber-mediated) client connections being mentioned, but not this.

I do not think this was announced. The only functionality drop should be no longer needed tftpsync and cobbler sync calls, which was quite a performance bottleneck.

Is there a way to reenable this through some changes to the Apache httpd configuration (since cobbler seems to try to use that pathway) or was the ability of cobbler to update the tftpd service data directory considered too much of a security vulnerability?

There is not.

Or is the issue that apache httpd and tftpd are in separate containers so that httpd can't access the tftpd data directories? Come to think that's last is almost certainly the problem, but shouldn't it be possible for the /var/lib/containers/storage/volumes/uyuni-proxy-tftpboot/ volume to be mapped into both containers? Looks like the z option is what should be used for sharing a volume - is that not working for some reason?

I expect that tftpboot volume will be removed in the future.

Both EFI and legacy pxe boot should work without any action from the user. How is your /srv/tftpboot/grub looks like on the server?

[ppanon2022]

On the server, it has some stuff in it.

uyuni-server:/srv/tftpboot # ls grub
aarch64_menu_items.cfg  images                              ppc_menu_items.cfg
arm_menu_items.cfg      local_efi.cfg                       s390_menu_items.cfg
bootia32.efi            local_efi.cfg.rpmsave               s390x_menu_items.cfg
grub.0                  local_legacy.cfg                    shim.efi
grub.cfg                local_legacy.cfg.rpmsave            system
grub.cfg.rpmsave        local_powerpc-ieee1275.cfg          system_link
grubx86.efi             local_powerpc-ieee1275.cfg.rpmsave  x86_64-efi
i386-pc                 ppc64_menu_items.cfg                x86_64_menu_items.cfg
i386_menu_items.cfg     ppc64el_menu_items.cfg
ia64_menu_items.cfg     ppc64le_menu_items.cfg

Is there any kind of access logging we can look at for the TFTP service? Looking in the /var/log of the tftpd container (with podman exec -ti uyuni-proxy-tftpd /bin/bash) doesn't show anything that seems relevant. The access_log for the httpd container is empty.

[aaannz]

This seems like a cobbler issue. I don't see grubx64.efi here. What do you have in /etc/cobbler/settings.yaml in the container? Check bootloader_formats key.
Seems like by default its grubx86.efi

Not sure why would cobbler ask for grubx64.efi in dhcp. In SUMA cobbler generated dhcp config is not supported. And I see that our documentation is lacking in this way.

So options:

• modify cobbler's /etc/cobbler/settings.yaml and change x86_64 loader to grubx64.efi, call cobbler mkloaders and cobbler sync
• modify dhcp template to use grubx86.efi

At my test systems both grubx86.efi and grubx64.efi are identical, so it is only what you prefer.

[ppanon2022]

We're mostly using the default cobbler settings I think. The parameters that we override are

default_name_servers_search: 
ldap_base_dn:
ldap_server: 
manage_dhcp: false
manage_dhcp_v4: false
next_server_v4: 
proxy_url_ext:
restart_dhcp: false
restart_dns: false
scm_track_author: cobbler <[email protected]>

That said, yes, the settings.yaml had

bootloaders_formats:
  IA64:
    binary_name: bootia64.efi
  aarch64:
    binary_name: grubaa64.efi
  arm:
    binary_name: bootarm.efi
  arm64-efi:
    binary_name: grubaa64.efi
    extra_modules:
    - efinet
  i386-efi:
    binary_name: bootia32.efi
  i386-pc-pxe:
    binary_name: grub.0
    extra_modules:
    - chain
    - pxe
    - biosdisk
    mod_dir: i386-pc
  i686:
    binary_name: bootia32.efi
  powerpc-ieee1275:
    binary_name: grub.ppc64le
    extra_modules:
    - net
    - ofnet
  x86_64-efi:
    binary_name: grubx86.efi
    extra_modules:
    - chain
    - efinet

However, changing it to use binary_name: grubx64.efi and running cobbler mkloaders did not create or add a grubx64.efi in the /srv/cobbler/grub directory so I don't think that would be an improvement. After those changes and commands the directory contents still look like this.

uyuni-server:/etc/pam.d # ls -al /srv/tftpboot/grub
total 3484
drwxr-xr-x.  6 root root    4096 Nov  4 23:47 .
drwxr-xr-x. 12 root root    4096 Sep 26 21:20 ..
-rw-r--r--.  1 root root       0 Nov  4 23:47 aarch64_menu_items.cfg
-rw-r--r--.  1 root root       0 Nov  4 23:47 arm_menu_items.cfg
-rw-r--r--.  1 root root  901120 Sep 25 14:16 bootia32.efi
-rw-r--r--.  1 root root  313077 Nov  4 23:43 grub.0
-rw-r--r--.  1 root root    4079 Jul 25 06:55 grub.cfg
-rw-r--r--.  1 root root    4079 May  3  2024 grub.cfg.rpmsave
-rw-r--r--.  1 root root 1310720 Nov  4 23:43 grubx86.efi
drwxr-xr-x.  2 root root   12288 Aug 23 12:30 i386-pc
-rw-r--r--.  1 root root       0 Nov  4 23:47 i386_menu_items.cfg
-rw-r--r--.  1 root root       0 Nov  4 23:47 ia64_menu_items.cfg
lrwxrwxrwx.  1 root root       9 Jul 14  2023 images -> ../images
-rw-r--r--.  1 root root    1371 Jul 25 06:55 local_efi.cfg
-rw-r--r--.  1 root root    1371 May  3  2024 local_efi.cfg.rpmsave
-rw-r--r--.  1 root root     152 Jul 25 06:55 local_legacy.cfg
-rw-r--r--.  1 root root     152 May  3  2024 local_legacy.cfg.rpmsave
-rw-r--r--.  1 root root     359 Jul 25 06:55 local_powerpc-ieee1275.cfg
-rw-r--r--.  1 root root     359 May  3  2024 local_powerpc-ieee1275.cfg.rpmsave
-rw-r--r--.  1 root root       0 Nov  4 23:47 ppc64_menu_items.cfg
-rw-r--r--.  1 root root       0 Nov  4 23:47 ppc64el_menu_items.cfg
-rw-r--r--.  1 root root       0 Nov  4 23:47 ppc64le_menu_items.cfg
-rw-r--r--.  1 root root       0 Nov  4 23:47 ppc_menu_items.cfg
-rw-r--r--.  1 root root       0 Nov  4 23:47 s390_menu_items.cfg
-rw-r--r--.  1 root root       0 Nov  4 23:47 s390x_menu_items.cfg
-rwxr-xr-x.  1 root root  965672 Apr 18  2024 shim.efi
drwxr-xr-x.  2 root root       6 Jul  2 04:55 system
drwxr-xr-x.  2 root root       6 Jul  2 04:55 system_link
drwxr-xr-x.  2 root root   12288 Aug 23 12:30 x86_64-efi
-rw-r--r--.  1 root root       0 Nov  4 23:47 x86_64_menu_items.cfg

Now the Uyuni documentation does say to only use pxelinux.0 as the filename option for DHCP, but I don't think that ever worked for us. The one time I actually managed to get a client to bootstrap grub before the container conversion, I'm pretty sure that wasn't with pxelinux.0, and it would not display OS options on the grub menu. The /srv/cobbler/pxelinux.cfg/ directory only has one file with basically an empty menu.

# more pxelinux.cfg/default
DEFAULT menu
PROMPT 0
MENU TITLE Cobbler | https://cobbler.github.io
TIMEOUT 200
TOTALTIMEOUT 6000
ONTIMEOUT local

LABEL local
        MENU LABEL (local)
        MENU DEFAULT
        LOCALBOOT -1

MENU end

Now I remember reading that cobbler should have both a distro and a profile set up, for it to work, and I only see a distro set up.

uyuni-server:/srv/tftpboot/grub # cobbler distro list
   Rocky8:1:SemtechCorporation
uyuni-server:/srv/tftpboot/grub # cobbler profile list

so I'm wondering if maybe something wasn't set up by Uyuni, but I thought I went through all the documented steps. Cobbler check gives

# cobbler check
The following are potential configuration items that you may want to fix:

1: some network boot-loaders are missing from /var/lib/cobbler/loaders. If you only want to handle x86/x86_64 netbooting, you may ensure that you have installed a *recent* version of the syslinux package installed and can ignore this message entirely. Files in this directory, should you want to support all architectures, should include pxelinux.0, andmenu.c32.
2: enable and start rsyncd.service with systemctl
3: reposync not installed, install yum-utils
4: yumdownloader is not installed, install yum-utils

Restart cobblerd and then run 'cobbler sync' to apply changes.

and andmenu.c32 is missing from the /srv/cobbler tree, but there's nothing there that seems to be something I can change in the container (or else it seems likely to have been replaced by proxy squid services and not be relevant). Has this functionality been successfully regression tested in the containerized environment with something other than SUSE SLES? Or is it like the AD/Kerberos sssd integration?

[aaannz]

However, changing it to use binary_name: grubx64.efi and running cobbler mkloaders did not create or add a grubx64.efi in the /srv/cobbler/grub directory so I don't think that would be an improvement. After those changes and commands the directory contents still look like this.

cobbler mkloaders only creates new loaders for cobbler internally. Then you need to call cobbler sync which creates /srv/tftpboot completely.

Now the Uyuni documentation does say to only use pxelinux.0 as the filename option for DHCP, but I don't think that ever worked for us.

pxelinux.0 should work, but do check if you are not reading saltboot/retail docs rather cobbler autoinstallation. saltboot used different path.
But anyway, please rather use grub.0 as pxelinux.0 is part of syslinux which is very outdated and on its wait out. I will need to adapt the docs.

Now I remember reading that cobbler should have both a distro and a profile set up, for it to work, and I only see a distro set up.

Indeed cobbler should have both distro and profile setup. Distro provides kernel, initrd. Profile provides autoyast or kickstart (or in case of containerized Uyuni, saltboot also uses cobbler underneath)