Migration Path For Replacing Mirrored Boot Drive

[skinlayers]

A little over a year ago, I setup a server using ubuntu_server_encrypted_root.sh (commit 558c6fe) with 2x Crucial MX500 1TB SSDs as a mirrored boot drive (sde & sdf), and 2x 14TB WD HDs as a mirrored encrypted datapool (sda & sdc). Now smartctl is showing the SSDs Percent_Lifetime_Remain at 9%, so I'd like to replace them with 2x Micron 5200MAX 960GB enterprise SSDs.

This brings up a few questions:

• What steps do I need to take to replace the encrypted mirrored SSDs one at a time? Can I just replace one, run zpool replace and let the mirror rebuild itself, or are there any additional steps that need to be taken to (for example) write zfsbootmenu to /boot/efi on the new drive?

• If I wanted to do a clean install of Ubuntu 22.04, is there anyway to add an existing encrypted datapool mirror afterwards without destroying it, so it seamlessly unlocks along with the root volume at boot? I know I would need to zfs export it first, but I don't know how to integrate the exported pool with a new install.

$ lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
[...]
sda         8:0    0  12.7T  0 disk
├─sda1      8:1    0  12.7T  0 part
└─sda9      8:9    0     8M  0 part
[...]
sdc         8:32   0  12.7T  0 disk
├─sdc1      8:33   0  12.7T  0 part
└─sdc9      8:41   0     8M  0 part
[...]
sde         8:64   0 931.5G  0 disk
├─sde1      8:65   0   512M  0 part  /boot/efi
├─sde2      8:66   0   500M  0 part
│ └─md127   9:127  0   499M  0 raid1
└─sde3      8:67   0 930.5G  0 part
sdf         8:80   0 931.5G  0 disk
├─sdf1      8:81   0   512M  0 part
├─sdf2      8:82   0   500M  0 part
│ └─md127   9:127  0   499M  0 raid1
└─sdf3      8:83   0 930.5G  0 part

$ sudo zpool list
NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool         928G  53.1G   875G        -         -    10%     5%  1.00x    ONLINE  -
datapool     12.7T  7.24T  5.47T        -         -     0%    56%  1.00x    ONLINE  -

Thank you!

[Sithuk]

Hi skinlayers,

1. I haven't tested your use case. However, I agree with your logic that in the scenario you outlined a mirror rebuild would only rebuild the zpool. "Zpool replace" wouldn't rebuild the boot or swap partitions on the mirror drive. You would have to partition the new mirror drive first. Then duplicate the boot partition contents to the new mirror drive boot partition. Then resliver the zpool ("zpool replace"). You will also need to update your mdraid array that is used for swap to remove the old drive, add the new one and re-sync.

2. I haven't tested your use case. However, you should be able to have an existing encrypted datapool mirror automatically unlock at boot following a clean root install. Have a look at the create_dpool_Func in the script. You will need to make sure your encryption passwords align on the new root pool and the datapool. You may need to check that the zfs cache has been updated after importing the datapool for the first time.

The most important advice I can give is to test in a VM first. Please come back and let me know how you got on and share any tips.

[skinlayers]

I'm attempting to test moving a mirrored datapool to a new zfs root install in Virtualbox, but I've hit a wall. After I export the datapool volume, I remove the mirrored root disk images, and replace them with 2x new ones. After booting from the Ubuntu 22.04.3 Desktop ISO, if I tried to run ./ubuntu_server_encrypted_root_zfs.sh install it would error out with:

efibootmgr: ** Warning ** : Boot000A has same label rEFInd Boot Manager Backup 1
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0005,0004,000A,0000,0001,0006,0007,0008,0002,0003
Boot0000* UiApp
Boot0001* UEFI VBOX CD-ROM VB2-01700376
Boot0002* UEFI VBOX HARDDISK VBab66dc93-ea5eb09e
Boot0003* UEFI VBOX HARDDISK VBab42738f-9b56a82c
Boot0004* rEFInd Boot Manager
Boot0006* UEFI PXEv4 (MAC:080027C1D395)
Boot0007* UEFI PXEv4 (MAC:08002791ED4A)
Boot0008  EFI Internal Shell
Boot000A* rEFInd Boot Manager Backup 1
Boot0005* rEFInd Boot Manager Backup 1
Invalid BootOrder order entry value0004,0005,0009

EFI boot order before initial zbm install

ubuntu@ubuntu:~$ sudo efibootmgr
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0000,0001,0002,0003,0004,0005,0006,0007,0008
Boot0000* UiApp
Boot0001* UEFI VBOX CD-ROM VB2-01700376
Boot0002* UEFI VBOX HARDDISK VB4b898ce8-fb835586
Boot0003* UEFI VBOX HARDDISK VB3b6c6d2f-fe957c42
Boot0004* UEFI VBOX HARDDISK VB63f95cee-4d05262c
Boot0005* UEFI VBOX HARDDISK VBf803dca9-c3a6e227
Boot0006* UEFI PXEv4 (MAC:080027C1D395)
Boot0007* UEFI PXEv4 (MAC:08002791ED4A)
Boot0008  EFI Internal Shell

I found I could get around this by using efibootmgr to remote the rEFInd Boot Manager entries before running the script.
Now the script is able to run to completion. I can reboot and unlock the rpool. However, once I reattach the 2x datapool disk images, I can no longer boot. rEFInd it appears to load the ubuntu kernel, but then I get:

emergency shell

type 'help' for online documentation
type 'exit' to return to ZFSBootMeny

ZFSBootMenu: environment file /zfsbootmenu/bootenvs is missing

zfsbootmenu / > _

If I remove the datapool disk images, I am able to boot the VM and unlock the rpool.

Any idea what I'm missing?

Thank you!

[skinlayers]

I believe the efibootmgr warning was the result of me trying to reuse the same VM with new boot VDIs. I decided to give it another shot, and I was finally able to get this working in VirtualBox. I'm not sure if all of these steps are required, but I've tested the procedure twice now.

Ubuntu Jammy (22.04) VM Setup

Edit: This is to recreated, in VirtualBox, the same OS, script version, and config that I used to setup my current home lab server. I'm probably being overly cautious by using jammy + an old version of the script to setup the jammy_datapool that I later attach to the noble VM, but I wanted to try to model my home setup as closely as possible to avoid potential data loss when attempting to migrate my existing encrypted datapool mirror to a fresh install, on new SSDs, of Ubuntu noble using the latest version of this script.

I started by setting up a new Ubuntu jammy VM.

• ISO Image: ubuntu-22.04.4-desktop-amd64.iso
• Base Memory: 8192 MB
• Processors: 2 CPUs
• Enable EFI
• 2x 25GB VDIs (for encrypted rpool mirror)
• 2x 50GB VDIs (for encrypted jammy_datapool mirror) in a folder separate from the VM e.g. VirtualBox VMs/zfs-data/

Note: I'm also using NAT Network + Port Forwarding rules to allow SSH from the host to the VM, but you could also use a Bridged Adapter (without Port Forwarding), or enable an additional interface and set it as a Host-only Adapter.

Boot the LiveCD, and select Try Ubuntu.
Open Terminal

sudo apt update
sudo apt install vim ssh git
sudo passwd ubuntu
ip a

Make note of the IP address. SSHing into the host allows you to copy and paste commands.
In the local Terminal, SSH to the VM:

ssh ubuntu@IP_ADDRESS

After connecting to the jammy VM via SSH:

git clone https://github.com/Sithuk/ubuntu-server-zfsbootmenu.git
cd ubuntu-server-zfsbootmenu
# Checking out the exact version of the script I used to setup my home lab jammy server.
git branch 2022-08-29 558c6fe
git switch 2022-08-29

Script settings changed from their defaults (jammy VM):

topology_root="mirror"
disks_root="2"
datapool="jammy_datapool"
topology_data="mirror"
disks_data="2"

Switch to the root user, and run the initial phase of the script.

sudo -i
cd /home/ubuntu
./ubuntu_server_encrypted_root_zfs.sh initial
# When prompted, select the 25GB /dev/sda and /dev/sdb for the rpool disks.

Reboot when initial install is complete.
Login as root (since this was setup with an old version of the script).
Run the postreboot and datapool phase of the script.

./ubuntu_server_encrypted_root_zfs.sh postreboot
./ubuntu_server_encrypted_root_zfs.sh datapool
# When prompted, select the 50GB /dev/sdc and /dev/sdd for the datapool disks.

Reboot.
Login as testuser and export jammy_datapool.

sudo zpool export jammy_datapool

Shutdown the jammy VM.
Detach the 2x 50GB datapool VDIs from the jammy VM.
Edit: It's not a bad idea to make a backup of these VDIs prior to attaching them to the noble we'll create below.

Ubuntu noble (24.04) VM Setup

Edit: This is to simulate installing Ubuntu noble with the latest version of the script on a new pair of encrypted SSDs, in preparation for later attaching the datapool we created as part of the jammy VM.

Create a new Ubuntu noble VM, with the same settings as the jammy VM, except:
Do NOT create 2x 50GB datapool VDIs, as we'll be attaching the ones we made previously for the jammy VM.
ISO Image: ubuntu-24.04-desktop-amd64.iso
Follow the same Terminal commands, until you have SSHed into the noble VM

After connecting to the VM via SSH:

git clone https://github.com/Sithuk/ubuntu-server-zfsbootmenu.git
cd ubuntu-server-zfsbootmenu

Script settings changed from their defaults (noble VM):

topology_root="mirror"
disks_root="2"

Continue following the same steps as for the jammy VM, until the initial install is done, and you've rebooted.
This time login as testuser.
Run the postreboot phase of the script

./ubuntu_server_encrypted_root_zfs.sh postreboot

Shutdown the VM
Now is a good time to make a snapshot of the noble VM.
Attach the 2x 50GB VDIs we created as part of the jammy VM.

How To Fix Booting noble VM with Preexisting datapool Attached

Edit: Boot the noble VM with the jammy_datapool we created as part of the jammy VM now attached.

Console output while booting:

Edit: zbm will fail to find any valid boot environments.

That's strange. It seems like maybe we're trying to boot off of the preexisting jammy_datapool we attached instead of rpool we just installed noble on to?

Force import rpool and then exit the recovery shell:

zfsbootmenu / > zpool import -f rpool
zfsbootmenu / > exit

---

You should now be prompted to unlock the rpool, and be able to select the noble boot environment:

Login as testuser.
Import jammy_datapool, update its keylocation, load the key, mount jammy_datapool, and regenerate ZBM:

sudo touch /etc/zfs/zfs-list.cache/jammy_datapool
sudo zpool import jammy_datapool
sudo zfs set keylocation=file:///etc/zfs/rpool.key jammy_datapool
sudo zfs load-key jammy_datapool
sudo zfs mount jammy_datapool
sudo generate-zbm

Reboot.

The VM now boots up correctly with the jammy_datapool attached.
I haven't attempted this outside of VirtualBox yet. Please let me know if I'm missing any steps.

[Sithuk]

Hi skinlayers and thank you for the feedback. I don't understand why the boot error following datapool creation would arise on a 24.04 system and not the 22.04 system. I'll see if I can duplicate your test results on the 24.04 system when I get some time for more testing.

[skinlayers]

The 22.04 system portion was just to setup, in VirtualBox, the same datapool configuration, using the same OS and script version, that I used on my Dell PowerEdge T420 home lab server, so I could test moving the datapool to a fresh install. I've ran into the same problem in the past when trying to move a datapool from a 22.04 VM to another 22.04 VM, before 24.04 had been released. So I don't think the OS version is the issue, but rather how VirtualBox + rEFInd + zbm handles having a pre-existing encrypted datapool attached. I've edited the text above to help clarify.

[skinlayers]

This weekend on my home lab server, I exported my 22.04 HDD datapool, installed new SSDs for the new rpool, and installed Ubuntu 24.04 on them (mirrored zfs with native encryption and remote access). When I reinserted the datapool drives into my server, I ran into the same "No boot environments with kernels found" issue that I encountered in VirtualBox, and was able to resolve it with the steps above.

[Sithuk]

Thank you for the update. Sounds like there is something not quite right. I'll have a look when I get some time for further development. Hopefully your feedback above will help anyone experiencing the issue.

[Sithuk]

It might be worth reporting the issue upsteam at zfsbootmenu. I'm not sure why zfsbootmenu didn't pickup your rpool.

[Sithuk]

Also, you could try setting the following as a kernel command line parameter in /boot/efi/EFI/ubuntu/refind_linux.conf

zbm.prefer=rpool

See https://docs.zfsbootmenu.org/en/v2.3.x/man/zfsbootmenu.7.html

[skinlayers]

Edit: Ignore this, as I made a typo when adding zbm.prefer=rpool to my kernel options.

I reverted my VirtualBox test VM back to before I attached the datapool, and made the change you suggested:

After making the edit, I ran sudo generated-zbm, shut down the VM, and attached the datapool.
However, I'm still running into the same issue when I start the VM with the datapool attached:

[Sithuk]

Your screenshot shows "zmb.prefer=rpool" not "zbm.prefer=rpool"?

[skinlayers]

Ah, good eye!
I just reverted my VirtualBox snapshot and tested it again, and zbm.prefer=rpool does seem to resolve the issue!
Thank you!

Side note: I'm still not sure why this was an issue on my home lab server, though, as the datapool's I moved is named vale and according to the docs for zbm.prefer, rpool should have been loaded before vale.

ZFSBootMenu will attempt to import as many pools as possible to identify boot environments and will, by default, look for the bootfs property on the first imported pool (sorted alphabetically) to select the default boot environment.[...]

[Sithuk]

Thanks skinlayers, good to read that you’ve got a solution. Your question about why zbm.prefer was needed seems a good one to raise as an issue on the zbm GitHub. If there is an issue, it’s always better to resolve it upstream.

I need to check whether this pool import issue is present with the default names of rpool and dpool. And what happens with different root pool and data pool names too. Including to duplicate your findings.