Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bazzite not updating grub after upgrade to 41 (stuck in 39) #2188

Closed
kikito opened this issue Jan 26, 2025 · 14 comments
Closed

Bazzite not updating grub after upgrade to 41 (stuck in 39) #2188

kikito opened this issue Jan 26, 2025 · 14 comments
Labels
bug Something isn't working

Comments

@kikito
Copy link

kikito commented Jan 26, 2025

Describe the bug

My system seems to be stuck in Bazzite 39, I can't seem to be able to upgrade to 41. I suspect a problem updating the grub menu.

What did you expect to happen?

I expected the Bazzite 41 option to appear on the Grub2 boot menu after updating, but it has not.

  • Discover says I am "Up to date"
  • I have also followed the upgrading guide. For good measure, I just executed ujust update on the console right now.
Output of `ujust upgrade`
── 10:40:30 - System update ────────────────────────────────────────────────────
Pulling manifest: ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
Checking out tree 5593736... done
No upgrade available.

── 10:40:37 - Distrobox ────────────────────────────────────────────────────────

── 10:40:37 - Firmware upgrades ────────────────────────────────────────────────
Metadata is up to date; use --force to refresh again.
Devices with no available firmware updates: 
 • BCM20702A0
 • System Firmware
 • UEFI Device Firmware
 • UEFI Device Firmware
 • WD BLACK SN770 1TB
Devices with the latest available firmware version:
 • UEFI dbx
No updates available

── 10:40:37 - Flatpak User Packages ────────────────────────────────────────────
Looking for updates…

Info: runtime org.kde.Platform branch 6.6 is end-of-life, with reason:
   We strongly recommend moving to the latest stable version of the Platform and SDK
Info: applications using this runtime:
   com.obsproject.Studio

Nothing to do.

── 10:40:38 - Flatpak System Packages ──────────────────────────────────────────
Looking for updates…

Info: app org.ryujinx.Ryujinx branch stable is end-of-life, with reason:
   This application is no longer maintained.

Nothing to do.

── 10:40:38 - Nix ──────────────────────────────────────────────────────────────
unpacking channels...

── 10:40:38 - pip3 ─────────────────────────────────────────────────────────────
Requirement already satisfied: pip in /home/kikito/.local/lib/python3.12/site-packages (24.3.1)

── 10:40:38 - Linux AMDGPU Control Application ─────────────────────────────────
Looking for updates...
LACT not installed, skipping.

── 10:40:39 - Mozilla GNOME Themes ─────────────────────────────────────────────
Looking for updates...

── 10:40:39 - Third Party CSS Loader Themes ────────────────────────────────────
Looking for updates...

── 10:40:39 - Summary ──────────────────────────────────────────────────────────
System update: OK
distrobox: OK
Firmware upgrades: OK
Flatpak: OK
nix: OK
pip3: OK
Linux AMDGPU Control Application: OK
Mozilla GNOME Themes: OK
Third Party CSS Loader Themes: OK

(R)eboot
(S)hell
(Q)uit

Pressing "quit" I get the message "topgrade finished successfully". I have rebooted afterwards. Grub2 only shows 2 fedora 39 options

I have looked at my /etc/grub.d/grub.cfg file. It does look like 41 is trying to source its config from a separate file (/boot/grub2/custom.cfg), while 39 puts its config menuitem options directly on the file:

Output of `sudo cat /boot/grub2/grub.cfg`
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -f ${config_directory}/grubenv ]; then
  load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
set tuned_initrd=""
### END /etc/grub.d/00_tuned ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/08_fallback_counting ###
insmod increment
# Check if boot_counter exists and boot_success=0 to activate this behaviour.
if [ -n "${boot_counter}" -a "${boot_success}" = "0" ]; then
  # if countdown has ended, choose to boot rollback deployment,
  # i.e. default=1 on OSTree-based systems.
  if  [ "${boot_counter}" = "0" -o "${boot_counter}" = "-1" ]; then
    set default=1
    set boot_counter=-1
  # otherwise decrement boot_counter
  else
    decrement boot_counter
  fi
  save_env boot_counter
fi
### END /etc/grub.d/08_fallback_counting ###

### BEGIN /etc/grub.d/10_linux ###
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 8324ea68-fada-463f-aaa0-a1dfa764c034
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=boot 813B-5EFC

# This section was generated by a script. Do not modify the generated file - all changes
# will be lost the next time file is regenerated. Instead edit the BootLoaderSpec files.
#
# The blscfg command parses the BootLoaderSpec files stored in /boot/loader/entries and
# populates the boot menu. Please refer to the Boot Loader Specification documentation
# for the files format: https://systemd.io/BOOT_LOADER_SPECIFICATION/.

# The kernelopts variable should be defined in the grubenv file. But to ensure that menu
# entries populated from BootLoaderSpec files that use this variable work correctly even
# without a grubenv file, define a fallback kernelopts variable if this has not been set.
#
# The kernelopts variable in the grubenv file can be modified using the grubby tool or by
# executing the grub2-mkconfig tool. For the latter, the values of the GRUB_CMDLINE_LINUX
# and GRUB_CMDLINE_LINUX_DEFAULT options from /etc/default/grub file are used to set both
# the kernelopts variable in the grubenv file and the fallback kernelopts variable.
if [ -z "${kernelopts}" ]; then
  set kernelopts="root=UUID=e8f0fac1-baf3-46f1-95bd-f848c3868da5 ro rootflags=subvol=root/ostree/deploy/default/deploy/9307d69d321f00528072b25fedcbc6c9862473df97a09f4a77a084ab7d410a4b.0 rd.luks.options=discard rhgb quiet "
fi

insmod blscfg
blscfg
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/10_reset_boot_success ###
# Hiding the menu is ok if last boot was ok or if this is a first boot attempt to boot the entry
if [ "${boot_success}" = "1" -o "${boot_indeterminate}" = "1" ]; then
  set menu_hide_ok=1
else
  set menu_hide_ok=0 
fi
# Reset boot_indeterminate after a successful boot
if [ "${boot_success}" = "1" ] ; then
  set boot_indeterminate=0
# Avoid boot_indeterminate causing the menu to be hidden more than once
elif [ "${boot_indeterminate}" = "1" ]; then
  set boot_indeterminate=2
fi
# Reset boot_success for current boot 
set boot_success=0
save_env boot_success boot_indeterminate
### END /etc/grub.d/10_reset_boot_success ###

### BEGIN /etc/grub.d/12_menu_auto_hide ###
if [ x$feature_timeout_style = xy ] ; then
  if [ "${menu_show_once}" ]; then
    unset menu_show_once
    save_env menu_show_once
    set timeout_style=menu
    set timeout=60
  elif [ "${menu_auto_hide}" -a "${menu_hide_ok}" = "1" ]; then
    set orig_timeout_style=${timeout_style}
    set orig_timeout=${timeout}
    if [ "${fastboot}" = "1" ]; then
      # timeout_style=menu + timeout=0 avoids the countdown code keypress check
      set timeout_style=menu
      set timeout=0
    else
      set timeout_style=hidden
      set timeout=1
    fi
  fi
fi
### END /etc/grub.d/12_menu_auto_hide ###

### BEGIN /etc/grub.d/14_menu_show_once ###
if [ x$feature_timeout_style = xy ]; then
  if [ "${menu_show_once_timeout}" ]; then
    set timeout_style=menu
    set timeout="${menu_show_once_timeout}"
    unset menu_show_once_timeout
    save_env menu_show_once_timeout
  fi
fi
### END /etc/grub.d/14_menu_show_once ###

### BEGIN /etc/grub.d/15_ostree ###
menuentry 'Fedora Linux 39.20240419.0 (Bazzite) (ostree:0)' --class gnu-linux --class gnu --class os --unrestricted 'ostree-0-8324ea68-fada-463f-aaa0-a1dfa764c034' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 8324ea68-fada-463f-aaa0-a1dfa764c034
linux16 /ostree/default-57f279771ffe042d1cc721e48fc86123e341e99345329eb0507eb3223ad0c94d/vmlinuz-6.7.12-202.fsync.fc39.x86_64 rd.luks.options=discard rhgb quiet root=UUID=e8f0fac1-baf3-46f1-95bd-f848c3868da5 rootflags=subvol=root rw ostree=/ostree/boot.1/default/57f279771ffe042d1cc721e48fc86123e341e99345329eb0507eb3223ad0c94d/0 gpu_sched.sched_policy=0
initrd16 /ostree/default-57f279771ffe042d1cc721e48fc86123e341e99345329eb0507eb3223ad0c94d/initramfs-6.7.12-202.fsync.fc39.x86_64.img
}
menuentry 'Fedora Linux 39.20240406.0 (Bazzite) (ostree:1)' --class gnu-linux --class gnu --class os --unrestricted 'ostree-1-8324ea68-fada-463f-aaa0-a1dfa764c034' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 8324ea68-fada-463f-aaa0-a1dfa764c034
linux16 /ostree/default-27fb5e27df7c4b0c0aa7ef3005c0a655dab79bdcb957011603058de0e1725c2e/vmlinuz-6.7.11-202.fsync.fc39.x86_64 rd.luks.options=discard rhgb quiet root=UUID=e8f0fac1-baf3-46f1-95bd-f848c3868da5 rootflags=subvol=root rw ostree=/ostree/boot.1/default/27fb5e27df7c4b0c0aa7ef3005c0a655dab79bdcb957011603058de0e1725c2e/0 gpu_sched.sched_policy=0
initrd16 /ostree/default-27fb5e27df7c4b0c0aa7ef3005c0a655dab79bdcb957011603058de0e1725c2e/initramfs-6.7.11-202.fsync.fc39.x86_64.img
}
### END /etc/grub.d/15_ostree ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
if [ "$grub_platform" = "efi" ]; then
        menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' {
                fwsetup
        }
fi
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/35_fwupd ###
### END /etc/grub.d/35_fwupd ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###

That seems to be part of the issue; assuming config_directory for grub means /boot/grub2, there's a /boot/grub2/grubenv file there, but there's no custom.cfg file:

sudo ls /boot/grub2
fonts  grub.cfg  grubenv

It definetly looks like there's a problem in writing to grub2, I got a warning when running rpm-ostree status (pasting that below, on the next section). It suggests running journalctl -b -1 -u ostree-finalize-staged.service, which I did:

Jan 25 23:18:15 fedora systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Jan 26 01:18:50 fedora systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Jan 26 01:18:50 fedora ostree[29050]: Finalizing staged deployment
Jan 26 01:18:51 fedora ostree[29050]: Copying /etc changes: 24 modified, 1 removed, 93 added
Jan 26 01:18:51 fedora ostree[29050]: Copying /etc changes: 24 modified, 1 removed, 93 added
Jan 26 01:18:51 fedora ostree[29050]: Refreshing SELinux policy
Jan 26 01:18:55 fedora ostree[29050]: Refreshed SELinux policy in 3581 ms
Jan 26 01:18:55 fedora ostree[29050]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jan 26 01:18:55 fedora ostree[29050]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jan 26 01:18:55 fedora ostree[29050]: Finalized deployment
Jan 26 01:18:55 fedora ostree[29050]: bootfs is sufficient for calculated new size: 165.8 MB
Jan 26 01:18:55 fedora ostree[29050]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jan 26 01:18:55 fedora ostree[29050]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jan 26 01:18:55 fedora ostree[29050]: error: Bootloader write config: grub2-mkconfig: Child process exited with code 127
Jan 26 01:18:55 fedora systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Jan 26 01:18:55 fedora systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Jan 26 01:18:55 fedora systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Jan 26 01:18:55 fedora systemd[1]: ostree-finalize-staged.service: Consumed 4.521s CPU time.

There is a couple warnings there about a version mismatch (10.42 vs 10.44) which I don't know what to do with.

Output of rpm-ostree status

ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
                   Digest: sha256:3120d11f0690f815432fe21f5d61f1b633e4e1a0a0fb22541b6ebde3f40a1246
                  Version: 41.20250124 (2025-01-24T08:14:07Z)
                     Diff: 2128 upgraded, 5 downgraded, 403 removed, 668 added
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf"' 

  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
                   Digest: sha256:fe357496bb5ac67edd223e8cf27ed8d29384eb6948a082b98c265ec8a9d9ac90
                  Version: 39.20240419.0 (2024-04-19T17:04:05Z)
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf"' 

● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
                   Digest: sha256:7c9d16a34d8192df6ba824198c2e2343faa8108d9b53e949cebeed4d849e2395
                  Version: 39.20240406.0 (2024-04-07T01:29:50Z)
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf"'

Hardware

This is a desktop PC.

  • CPU: AMD Ryzen 5 7600X 6-Core
  • Motherboard: Asus TUF GAMING B650-PLUS
  • Graphics card: Radeon RX 6650 XT Core 8GB GDDR6
  • Hard Drive: WD_BLACK SN770 1TB

Extra information or context

No response

@dosubot dosubot bot added the bug Something isn't working label Jan 26, 2025
@jbtrystram
Copy link

Hello !
Can you show me the output of the following :
findmnt / and rpm-ostree version ?

And also :
journalctl -u ostree-finalize-staged.service --no-pager -o cat -b -1

@kikito
Copy link
Author

kikito commented Jan 29, 2025

Hi @jbtrystram , right away:

Output of findmt /:

$ findmt /

TARGET
  SOURCE                                                                                                                FSTYPE OPTIONS
/ /dev/nvme0n1p3[/root/ostree/deploy/default/deploy/9307d69d321f00528072b25fedcbc6c9862473df97a09f4a77a084ab7d410a4b.0] btrfs  rw,noatime,seclabel,compress-force=zstd:1,ssd,discard=async,space_cache=v2,commit=120,subvolid=257,subvol=/root

rpm-ostree version said unknown command, I suppose you meant rpm-ostree --version. Here it is:

$ rpm-ostree --version

rpm-ostree:
 Version: '2024.4'
 Git: 900d805f2116135469f31931250a4d14e2be44bb
 Features:
  - rust
  - compose
  - container
  - fedora-integration

And finally, here's journalctl:

$ journalctl -u ostree-finalize-staged.service --no-pager -o cat -b -1

Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Finalizing staged deployment
Copying /etc changes: 24 modified, 1 removed, 93 added
Copying /etc changes: 24 modified, 1 removed, 93 added
Refreshing SELinux policy
Refreshed SELinux policy in 3587 ms
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Finalized deployment
bootfs is sufficient for calculated new size: 165.8 MB
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
error: Bootloader write config: grub2-mkconfig: Child process exited with code 127
ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
ostree-finalize-staged.service: Failed with result 'exit-code'.
Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
ostree-finalize-staged.service: Consumed 4.536s CPU time.

I hope it helps, let me know if there's anything else I can provide. Thanks for your help!

@HikariKnight
Copy link
Member

since youre updating from fedora 39 to fedora 41, did you read https://universal-blue.discourse.group/t/important-announcement-regarding-system-updates-action-needed/2689 ?

rebase to a bazzite version based on fedora 40 before july 2024 then run the script from the announcement, then rebase to latest and it should work possibly.
not sure about the mismatches you are getting though

@bronxct1
Copy link

I am having this same issue coming from 40. Trying to rebase with stable says I'm up to date but I have no option to boot into 41.

findmnt / output:

$ findmnt /
TARGET SOURCE                                                                                                                FSTYPE OPTIONS
/      /dev/nvme1n1p3[/root/ostree/deploy/default/deploy/7e9d28c96bdd06ad30eecaeedc3da29768629a9686d2ef81a9e800c0aa9c021d.0] btrfs  rw,noatime,seclabel,compress-force=zstd:1,ssd,discard=async,space_cache=v2,commit=120,subvolid=258,subvol=/root

rpm-ostree --version

$ rpm-ostree --version
rpm-ostree:
 Version: '2024.6'
 Git: 1dda51b264eec8003eb6032f1f41844754ec163b
 Features:
  - rust
  - compose
  - container
  - fedora-integration

journalctl

Journal file /var/log/journal/79d003c8d6a84307a0a32ada0fcc8aa2/system@00061fd348d6b34b-24912c90cee5aeed.journal~ is truncated, ignoring file.

@kikito
Copy link
Author

kikito commented Feb 1, 2025

@HikariKnight thank you for your response. I am not an expert user of fedora/bazzite. I have used ubuntu in the past. I don't know that "rebasing" does. It must be what this describes: https://docs.bazzite.gg/Installing_and_Managing_Software/Updates_Rollbacks_and_Rebasing/rebase_guide/

I don't seem to have bazzite-rollback-helper installed (perhaps it was added in more recent versions of bazzite?) so I guess I need to go the manual way.

However, I don't know how to do this part:

rebase to a bazzite version based on fedora 40 before july 2024

When I try to list the parts available with skopeo (skopeo list-tags docker://ghcr.io/ublue-os/bazzite | sort -rV), all I see are builds for 41. Are older builds available somewhere else?

@HikariKnight
Copy link
Member

@HikariKnight thank you for your response. I am not an expert user of fedora/bazzite. I have used ubuntu in the past. I don't know that "rebasing" does. It must be what this describes: https://docs.bazzite.gg/Installing_and_Managing_Software/Updates_Rollbacks_and_Rebasing/rebase_guide/

I don't seem to have bazzite-rollback-helper installed (perhaps it was added in more recent versions of bazzite?) so I guess I need to go the manual way.

However, I don't know how to do this part:

rebase to a bazzite version based on fedora 40 before july 2024

When I try to list the parts available with skopeo (skopeo list-tags docker://ghcr.io/ublue-os/bazzite | sort -rV), all I see are builds for 41. Are older builds available somewhere else?

actually run the script from the link i sent then run
rpm-ostree rebase ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:stable-41.20250124
then reboot and if it boots fine then run
rpm-ostree rebase ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:stable

@kikito
Copy link
Author

kikito commented Feb 2, 2025

actually run the script from the link i sent then run rpm-ostree rebase ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:stable-41.20250124 then reboot and if it boots fine then run rpm-ostree rebase ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:stable

Hi @HikariKnight . I tried running the script you linked (curl -sL https://fix.universal-blue.org/ | sudo bash ) which did a bunch of stuff and prompted me to reboot, but I didn't. As instructed I ran:

rpm-ostree rebase ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:stable-41.20250124

Which also prompted me to reboot, so I did.

Question: am I supposed to see any difference in grub by this point? I still see the same options I saw before (only options for fedora 39 appear).

I could reboot with Fedora 39, but I don't think that is what you were suggesting. I think you were expecting for 41 to pop up in the grub menu by this point, which it hasn't.

Strongly considering doing a full reinstall at this point.

@HikariKnight
Copy link
Member

it should update you to fedora 41 as i did a large gap update like that myself and it worked so this is very odd

@atimeofday
Copy link

(curl -sL https://fix.universal-blue.org/ | sudo bash ) which did a bunch of stuff and prompted me to reboot, but I didn't

@HikariKnight Is the order of operations here important, such as rebooting twice, after the fix script and then after the rebase?

@kikito What is the output of rpm-ostree status?

@kikito
Copy link
Author

kikito commented Feb 4, 2025

@atimeofday same as before:

$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         error: Bootloader write config: grub2-mkconfig: Child process exited with code 127
         check `journalctl -b -1 -u ostree-finalize-staged.service`
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
                   Digest: sha256:fe357496bb5ac67edd223e8cf27ed8d29384eb6948a082b98c265ec8a9d9ac90
                  Version: 39.20240419.0 (2024-04-19T17:04:05Z)
                     Diff: 146 upgraded, 26 removed, 17 added
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf"' 

● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
                   Digest: sha256:7c9d16a34d8192df6ba824198c2e2343faa8108d9b53e949cebeed4d849e2395
                  Version: 39.20240406.0 (2024-04-07T01:29:50Z)
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf"'

Output of journalctl -b -1 -u ostree-finalize-staged.service for completeness is also similar to the one I put in the opening post:

$ journalctl -b -1 -u ostree-finalize-staged.service
Feb 03 22:47:39 fedora systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Feb 03 23:30:36 fedora systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Feb 03 23:30:36 fedora ostree[30012]: Finalizing staged deployment
Feb 03 23:30:37 fedora ostree[30012]: Copying /etc changes: 24 modified, 1 removed, 93 added
Feb 03 23:30:37 fedora ostree[30012]: Copying /etc changes: 24 modified, 1 removed, 93 added
Feb 03 23:30:37 fedora ostree[30012]: Refreshing SELinux policy
Feb 03 23:30:41 fedora ostree[30012]: Refreshed SELinux policy in 3674 ms
Feb 03 23:30:41 fedora ostree[30012]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Feb 03 23:30:41 fedora ostree[30012]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Feb 03 23:30:41 fedora ostree[30012]: Finalized deployment
Feb 03 23:30:41 fedora ostree[30012]: bootfs is sufficient for calculated new size: 165.8 MB
Feb 03 23:30:41 fedora ostree[30012]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Feb 03 23:30:41 fedora ostree[30012]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Feb 03 23:30:41 fedora ostree[30012]: error: Bootloader write config: grub2-mkconfig: Child process exited with code 127
Feb 03 23:30:41 fedora systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Feb 03 23:30:41 fedora systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Feb 03 23:30:41 fedora systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Feb 03 23:30:41 fedora systemd[1]: ostree-finalize-staged.service: Consumed 4.540s CPU time.

@kikito
Copy link
Author

kikito commented Feb 4, 2025

Googling around I found an issue similar to mine (coreos/rpm-ostree#3715 )

One of the diagnosis steps is running OSTREE_DEBUG_GRUB2=1 ostree admin finalize-staged from a root (sudo -s) terminal:

$ sudo -s
[sudo] password for xxx: yyy

(root)$  OSTREE_DEBUG_GRUB2=1 ostree admin finalize-staged

Copying /etc changes: 24 modified, 1 removed, 93 added
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
bootfs is sufficient for calculated new size: 165.8 MB
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
/etc/default/grub: line 62: terminal_output: command not found
error: Bootloader write config: grub2-mkconfig: Child process exited with code 127

That is interesting, it says that /etc/default/grub contains something it doesn't understand. terminal_output is unknown.

I then found this stackoverflow question (https://unix.stackexchange.com/questions/434446/etc-default-grub-line-60-terminal-output-command-not-found) from someone having a similar issue. Their issue was:

By mistake I ran this command grub2-mkconfig -o /etc/default/grub this replaced /etc/default/grub file with /boot/grub2/grub.cfg

Which might be part of my problem:

$ diff /etc/default/grub /boot/grub2/grub.cfg
(empty)

Those two files are identical in my machine at the moment. On that stackoverflow question /etc/default/grub needed to be something completely different.

What is /etc/default/grub supposed to look like?

@atimeofday
Copy link

What is /etc/default/grub supposed to look like?

Mine is edited for theming but should provide a ballpark estimate of what it should look like:

GRUB_TIMEOUT=3
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_CMDLINE_LINUX="rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_TERMINAL_OUTPUT="gfxterm"
GRUB_THEME="/boot/grub2/themes/Sekiro/theme.txt"
GRUB_TIMEOUT_STYLE=menu

Which uhhh.... yeah that looks nothing like /boot/grub2/grub.cfg. I can see why something would have exploded, so to speak.

Based on my old notes here https://universal-blue.discourse.group/t/grub-theming-guide-for-silverblue-ublue/370 the original file was:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_CMDLINE_LINUX="rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_TERMINAL_OUTPUT="console"

The BLSCFG line may also be altered, but should theoretically only have the cosmetic effect of keeping Grub from displaying duplicate entries on Fedora Atomic images.

@kikito
Copy link
Author

kikito commented Feb 6, 2025

This did the thing! I was able to finally update to 41. Putting the steps here in case it's useful for someone in the future.

Note that these steps are from memory so some of them might not be exact/correct/necessary:

  1. Copy-pasted my /etc/default/grub file from the one @atimeofday put above (thank you thank you!)
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_CMDLINE_LINUX="rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_TERMINAL_OUTPUT="console"
  1. Ran sudo grub2-mkconfig -o /boot/grub2/grub.cfg to install the config file. Saw no errors

  2. Ran ostree admin finalize-staged, saw no errors:

$ sudo -s
[sudo] password for xxx: yyy

(root)$  OSTREE_DEBUG_GRUB2=1 ostree admin finalize-staged
  1. Rebooted. I still saw 39 on the grub menu

  2. Installed the fix:

curl -sL https://fix.universal-blue.org/ | sudo bash
  1. Then, without rebooting, rebased to an older 41:
rpm-ostree rebase ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:stable-41.20250124
  1. Rebooted again. I could see 41 in the grub menu at this point! Rebooted on 41
  2. Rebased to the latest 41 stable:
rpm-ostree rebase ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:stable
  1. Another reboot, and I am now in 41!

The TLDR is: something had screwed with my /etc/default/grub file, preventing further updates to grub. Restoring that file to a sane default was the first fix. Then I had to apply the universal blue fix and update to 41 in 2 steps as @HikariKnight recommended.

Retrospectively, a lot of these issues could have been detected sooner if grub was slightly more verbose on its error messages. All I was getting was a Regex version mismatch when in reality the error message that would have helped was terminal_output: command not found. But in order to get that error one has to run as root with OSTREE_DEBUG_GRUB2=1. Perhaps this should be active by default.

For now, closing this issue. Thanks everyone for the help.

@kikito kikito closed this as completed Feb 6, 2025
@jbtrystram
Copy link

Thanks for the detailed write-up @kikito !

FYI (this may be useful to someone else finding this), a copy of the default /etc files lives under /usr/etc in any rpm-ostree distribution. So you could find the default /etc/default/grub file contents in there.
In this case it's a bit weird, as the file is not there, I am not sure why. I'm going to ask around to learn more :)

I recommend this read about /etc/ in rpm-ostree: https://blog.verbum.org/2014/01/24/why-ostree-requires-usretc/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants