-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mount /usr: Implement try-sleep loop (SAN + FC) #1218
Conversation
Thank you for contributing to the Leapp project!Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. If you need a different version of leapp, e.g. from PR#42, use It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported,
See other labels for particular jobs defined in the Please open ticket in case you experience technical problem with the CI. (RH internal only) Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra. |
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
3242d97
to
c0c7e0c
Compare
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Fixed
Show fixed
Hide fixed
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Outdated
Show resolved
Hide resolved
...pgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/mount_usr.sh
Outdated
Show resolved
Hide resolved
…ation This problem is typical for SAN + FC when the storage needs sometimes more time for the initialisation. Implemented try-sleep loop. Retry the activation of the storage + /usr mounting in 15s. The loop can be repeated 10 times, so total time is 150s right now for the activation. Note that this is not proper solution for the storage initialisation, however we have discovered some obstacles in the bootup process to be able to do it correctly as we would like to. Regarding limited time, we are going to deliver this solution, that should improve the experience and should be safe to not cause regressions for already working functionality. We expect to provide better solution for newer upgrades paths in future (IPU 8->9 and newer). jira: https://issues.redhat.com/browse/RHEL-3344
Manual testing on my LVM VMs:
Lsblk from tested machines (used multiple VGs):
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code looks OK, I've tried upgrading with the patch myself using a simple setup. As the results on more complex LVM setups conducted by @pirat89 we positive, I am merging this.
Sorry I don't like that proposal much, because it happens the |
Based on feedback from @rmetrich¹ the sleep period should be applied always as the problem with initialisation is happening also on systems with higher than few number of LVs where /usr is not on dedicated volume. 1: oamg#1218 (comment)
Based on feedback from @rmetrich¹ the sleep period should be applied always as the problem with initialisation is happening also on systems with higher than few number of LVs where /usr is not on dedicated volume. 1: oamg#1218 (comment)
Based on feedback from @rmetrich¹ the sleep period should be applied always as the problem with initialisation is happening also on systems with higher than few number of LVs where /usr is not on dedicated volume. 1: #1218 (comment)
Based on feedback from @rmetrich¹ the sleep period should be applied always as the problem with initialisation is happening also on systems with higher than few number of LVs where /usr is not on dedicated volume. 1: oamg#1218 (comment) (cherry picked from commit da5ce33)
## Packaging - .. names of packages, dependencies, changes in provided capabilities.... ## Upgrade handling ### Fixes - Add missing RHUI GCP config info for RHEL for SAP (oamg#1253) - Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210) - Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212) - Fix evaluation of PES data (oamg#1194) - Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169) - Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240) - Fix the parsing of the lscpu output (oamg#1184, oamg#1208) - Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178) - Fix upgrade on aarch64 via RHUI on AWS (oamg#1240) - Handle a false positive GPG check error when TargetUserSpaceInfo is missing (oamg#1269) - Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205) - Update the default kernel cmdline (oamg#1193, oamg#1216) - Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211) - Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219) - [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243) - [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260) - [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270) ### Enhancements - [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246) - Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199) - Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197) - Improve set_systemd_services_states logging (oamg#1213) - Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152) - Introduce experimental upgrades in 'live' mode for the testing (oamg#1248) - Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241) - Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274) - Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242) - Sort lists in dnf-plugin-data for easier overview (oamg#1231) - [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198) - [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206) - [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195) - [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249) - [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181) - [IPU 9 -> 10] Introduced number of changes to enable experimental IPU 9 -> 10 (oamg#1169) - [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268) - [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273) ## Additional changes interesting for devels - drop unused `packager` field from gpg-signatures.json (oamg#1233) - [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible - [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo
## Packaging - Start building for EL 9 in the upstream repository on COPR (#1169) ## Upgrade handling ### Fixes - Add missing RHUI GCP config info for RHEL for SAP (#1253) - Fix creation of the post upgrade report about changes in states of systemd services (#1210) - Fix detection of valid sshd config with internal-sftp subsystem in Leapp (#1212) - Fix evaluation of PES data (#1194) - Fix failing "update-ca-trust" command caused by missing util-linux package (#1169) - Fix handling of versions in RHUI configuration for ELS and SAP upgrades (#1240) - Fix the parsing of the lscpu output (#1184, #1208) - Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (#1178) - Fix upgrade on aarch64 via RHUI on AWS (#1240) - Handle a false positive GPG check error when TargetUserSpaceInfo is missing (#1269) - Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (#1205) - Update the default kernel cmdline (#1193, #1216) - Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (#1211) - Wait for the storage initialization when /usr is on separate file system - covering SAN (#1218, #1219) - [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (#1243) - [IPU 7 -> 8] Fix detection of bootable device on RAID (#1260) - [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (#1270) ### Enhancements - [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (#1245, #1246) - Update leapp data files (#1280) - Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (#1199) - Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (#1197) - Improve set_systemd_services_states logging (#1213) - Include leapp command execution and defined leapp envars inside leapp.db - (#1152) - Introduce experimental upgrades in 'live' mode for the testing (#1248) - Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (#1241) - Several minor improvements in messages printed in console output (#1173, #1214, #1274) - Several minor improvements in report and error messages (#1207, #1217, #1234, #1235, #1242) - Sort lists in dnf-plugin-data for easier overview (#1231) - [IPU 7 -> 8] Allow upgrade of content from ELS repositories (#1198) - [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (#1206) - [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (#1195) - [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (#1249) - [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (#1181) - [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (#1169) - [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (#1268) - [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (#1169, #1273) ## Additional changes interesting for devels - drop unused `packager` field from gpg-signatures.json (#1233) - [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible - [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo
## Packaging - Start building for EL 9 in the upstream repository on COPR (oamg#1169) ## Upgrade handling ### Fixes - Add missing RHUI GCP config info for RHEL for SAP (oamg#1253) - Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210) - Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212) - Fix evaluation of PES data (oamg#1194) - Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169) - Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240) - Fix the parsing of the lscpu output (oamg#1184, oamg#1208) - Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178) - Fix upgrade on aarch64 via RHUI on AWS (oamg#1240) - Handle a false positive GPG check error when TargetUserSpaceInfo is missing (oamg#1269) - Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205) - Update the default kernel cmdline (oamg#1193, oamg#1216) - Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211) - Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219) - [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243) - [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260) - [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270) ### Enhancements - [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246) - Update leapp data files (oamg#1280) - Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199) - Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197) - Improve set_systemd_services_states logging (oamg#1213) - Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152) - Introduce experimental upgrades in 'live' mode for the testing (oamg#1248) - Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241) - Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274) - Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242) - Sort lists in dnf-plugin-data for easier overview (oamg#1231) - [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198) - [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206) - [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195) - [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249) - [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181) - [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (oamg#1169) - [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268) - [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273) ## Additional changes interesting for devels - drop unused `packager` field from gpg-signatures.json (oamg#1233) - [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible - [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo (cherry picked from commit 03c257b)
Based on feedback from @rmetrich¹ the sleep period should be applied always as the problem with initialisation is happening also on systems with higher than few number of LVs where /usr is not on dedicated volume. 1: oamg#1218 (comment) (cherry picked from commit da5ce33)
## Packaging - Start building for EL 9 in the upstream repository on COPR (oamg#1169) ## Upgrade handling ### Fixes - Add missing RHUI GCP config info for RHEL for SAP (oamg#1253) - Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210) - Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212) - Fix evaluation of PES data (oamg#1194) - Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169) - Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240) - Fix the parsing of the lscpu output (oamg#1184, oamg#1208) - Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178) - Fix upgrade on aarch64 via RHUI on AWS (oamg#1240) - Handle a false positive GPG check error when TargetUserSpaceInfo is missing (oamg#1269) - Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205) - Update the default kernel cmdline (oamg#1193, oamg#1216) - Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211) - Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219) - [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243) - [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260) - [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270) ### Enhancements - [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246) - Update leapp data files (oamg#1280) - Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199) - Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197) - Improve set_systemd_services_states logging (oamg#1213) - Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152) - Introduce experimental upgrades in 'live' mode for the testing (oamg#1248) - Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241) - Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274) - Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242) - Sort lists in dnf-plugin-data for easier overview (oamg#1231) - [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198) - [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206) - [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195) - [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249) - [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181) - [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (oamg#1169) - [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268) - [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273) ## Additional changes interesting for devels - drop unused `packager` field from gpg-signatures.json (oamg#1233) - [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible - [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo (cherry picked from commit 03c257b)
This problem is typical for SAN + FC when the storage needs sometimes more time for the initialisation. Implemented try-sleep loop. Retry the activation of the storage + /usr mounting in 15s. The loop can be repeated 10 times, so total time is 150s right now for the activation.
Note that this is not proper solution for the storage initialisation, however we have discovered some obstacles in the bootup process to be able to do it correctly as we would like to. Regarding limited time, we are going to deliver this solution, that should improve the experience and should be safe to not cause regressions for already working functionality. We expect to provide better solution for newer upgrades paths in future (IPU 8->9 and newer)
jira: https://issues.redhat.com/browse/RHEL-3344
Note the related PR where we will try to address this problem properly will be most likely #1202