Special kernel modules

[markohrastovec]

Hi,

does the switch to Almalinux preserve custom kernel modules? We have installed CentOS8 on an IBM server, which requires additional dd-megaraid_sas and dd-mpt3sas. They have unfortunately been removed from RedHat 8:

• https://access.redhat.com/discussions/3722151
• http://elrepoproject.blogspot.com/2019/08/rhel-80-and-support-for-removed-adapters.html
• http://linux-mirrors.fnal.gov/linux/elrepo/dud/el8/x86_64/

We have added these during installation in order for installation to recognize the disks. In first versions of CentOS 8 there was a bug, which did not upgrade kernel properly. These modules were removed on new kernel install and disks were not recognized any more. Later, that bug was fixed, and now updates are safe. We are afraid to make a switch and install Almalinux kernel. I checked the script. It installs new kernel, but I am not sure if the added modules will be preserved.

Does anybody have information, about that? I guess, when I have time and resources, I will have to test that sometime in the future, if I want to switch to Almalinux.

[cschug]

I have not really interest to ran AlmaLinux on very dated hardware but as I had also an old, unused server around with an unsupported RAID controller, I just did some test installations just for fun.

The hardware is a Dell PowerEdge R610 (firmware release 6.6.0) with a Dell PERC 6/i Integrated RAID Controller (LSI 1078 chipset which requires the megaraid_sas driver).

# grep ^ /sys/devices/virtual/dmi/id/{sys_vendor,product_name,bios_{vendor,version},board_name}
/sys/devices/virtual/dmi/id/sys_vendor:Dell Inc.
/sys/devices/virtual/dmi/id/product_name:PowerEdge R610
/sys/devices/virtual/dmi/id/bios_vendor:Dell Inc.
/sys/devices/virtual/dmi/id/bios_version:6.6.0
/sys/devices/virtual/dmi/id/board_name:086HF8

# lspci -mmnnvk -d ::0104
Slot:   03:00.0
Class:  RAID bus controller [0104]
Vendor: Broadcom / LSI [1000]
Device: MegaRAID SAS 1078 [0060]
SVendor:        Dell [1028]
SDevice:        PERC 6/i Integrated RAID Controller [1f0c]
Rev:    04
Driver: megaraid_sas
Module: megaraid_sas

Some words in my installation infrastructure: servers perform a network boot (PXE on older hardware like this Dell box, and HTTP boot on more recent hardware like HPE ProlIant Gen10 and higher) with a custom iPXE as NBP. The iPXE configuration enriches the boot information like which operating system should be installed (AlmaLinux or CentOS for example) and which extra kernel boot parameters are required for that hardware. For any EL8.3-based distributions that's the place where the ELrepo driver ISO is being added like

inst.dd=http://mirror.example.org/elrepo/dud/el8/x86_64/dd-megaraid_sas-07.714.04.00-1.el8_3.elrepo.iso

for any hardware requiring this driver (with mirror.example.org being the redacted name of my local mirror). The information whether AlmaLinux or CentOS should be installed is being added as a custom kernel parameter as well because for anything which smells like EL8 the very same Kickstart configuration is being used and there are only a very few conditionals based on this kernel parameter by extracting it from /proc/cmdline, for example which RPM repositories should be used. So it's one kickstart to rule them all, regardless of being AlmaLinux, CentOS, baremetal, VMs or Vagrant Base Boxes, therefore I can exclude any major differences or side effects by the configuration of the installation. iPXE finally chains the corresponding initrd.img and vmlinuz files and boots into the installation.

I did four installation scenarios:

• Server configured to run in UEFI mode
  • Initial installation with AlmaLinux 8: broken
  • Initial installation with CentOS 8, then running the migration script almalinux-deploy.sh: broken
• Server configured to run in BIOS mode
  • Initial installation with AlmaLinux 8: works
  • Initial installation with CentOS 8, then running the migration script almalinux-deploy.sh: works

In the UEFI case, while the installation with CentOS 8 works and results in a bootable and working system, it does not for AlmaLinux 8. The AlmaLinux installation picks up the driver and also performs the actuall installation (creates file systems, installs packages, etc.). Once the installation is done and the server reboots it looks like it cannot find grub2 (black screen, just a blinking cursor, no error message). The same with running the migration on a CentOS 8 system. All looks fine, but after reboot the machine is stuck in the same way (and yes, I did take care to provide the UEFI boot entry for AlmaLinux manually as this is not handled yet by the migration script, see #31). Something seems to be missing/different to CentOS 8 here but I didn't debug it any further as I personally have no interest in it.

Results might be different on different hardware but I would strongly suggest to do your own testing.I would consider it very too risky to run any production workload on very dated, unsupported hardware. YMMV

[markohrastovec]

Thanks! That is very useful information.

Just for info. We have tested it with Oracle Linux 8, and it works (with UEFI). There seems to be no difference between CentOS and Oracle in installing and using ISOs with custom drivers (at least megaraid_sas and mpt3sas).

[theconnstruct]

@cschug Out of curiosity, would you be able to dump the UEFI boot entries for CentOS 8 before the migration script and after?

# efibootmgr -v

[markohrastovec]

@conws My dumped UEFI boot entries are shown below. For now I do not dare to switch to AlmaLinux, as I think it will not work according to @cschug's report. Maybe @cschug can provide more info. I certainly can't as I am pretty sure for now that I would not be able to run "efibootmgr" after switch. :(

BootCurrent: 000E
Timeout: 10 seconds
BootOrder: 000E,0002,0000,0001,0003,0004,000C,0006,0007,0008,0009,000A,000B,0005,000D
Boot0000* CD/DVD Rom	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,0c)
Boot0001* Floppy Disk	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,00)
Boot0002* Hard Disk 0	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,08)
Boot0003* PXE Network	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,06)
Boot0004* Hard Disk 1	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,09)
Boot0005* Hard Disk 2	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,0a)
Boot0006* Hard Disk 3	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,0b)
Boot0007* Hard Disk 4	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,0e)
Boot0008* USB Storage	PciRoot(0x0)/Pci(0x1a,0x7)/USB(1,0)/HD(1,MBR,0xfccb5eaf,0xed00,0x39b1300)
Boot0009* Diagnostics	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,da)
Boot000A* iSCSI	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,04)
Boot000B* iSCSI Critical	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,05)
Boot000C* Legacy Only	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,ee)
Boot000D* Embedded Hypervisor	VenMedia(0c588db8-6af4-11dd-a992-00197d890238,01)
Boot000E* CentOS Linux	HD(1,GPT,3cdeea6f-95f4-4e73-878f-9878a02dd9b3,0x800,0xfa000)/File(\EFI\centos\shimx64.efi)

P.S.
Now that I am looking at the entries, I see, they are of no use unless there are also the entries after the switch to AlmaLinux. Sorry. I do not have those for now.

[cschug]

@conws regarding this general UEFI boot entry issues, I wrote all relevant information (before and after) in this comment. It wasn't any different on that Dell R610.

My usual workaround is to add the AlmaLinux boot entry on my own (and delete the obsolete CentOS one). This should normally do. It worked on any UEFI machine so far which has supported hardware without the need for additional ELrepo drivers.
It just doesn't with the ELrepo drivers and using otherwise unsupported hardware being the obvious difference, that doesn't mean ELrepo drivers are causing the issue, the reason is unknown so far.