Update Orion build to Rocky 9 #754
Comments
|
@DavidHuber-NOAA , what do we know about the Orion Rocky 9 transition for spack-stack? Will Orion Rocky 9 builds use the same spack-stack as Hercules Rocky 9? |
|
@RussTreadon-NOAA Spack-stack will be rebuilt for Orion after the upgrade to Rocky 9. I'm not sure if we will need to update our paths in any of the module files, but we will need to update compiler version numbers. Also, I suspect that we will need to |
|
It looks like the spack-stack installs will be tracked by JCSDA/spack-stack#981. |
|
Thank you @DavidHuber-NOAA for the detailed information. We will track spack-stack #981. |
|
I updated the ORION modules for GSI The GSI compiled successfully with the updated modules. |
|
Thank you @emilyhcliu for your note. We updated |
This reverts commit 086e273.
This reverts commit aab4a06.
|
@TingLei-NOAA , given the Hercules HelpDesk inquiry (RDHPCS#2024020854000112) would you be willing to update Emily provides a link above to her branch with Orion Rocky 9 updates. Click here to see exactly what she changed. |
|
@RussTreadon-NOAA Thanks. Ok I will follow the link you gave pointing to Emily's branch to have a working copy . |
|
An update: started from Emily's branch , first test the modules of https://github.com/emilyhcliu/GSI/blob/feature/orion_modules/modulefiles/gsi_orion.intel.lua. It seems something with stack-spack is not setup correctly somewhere. I had opened a ticket with orion helpdesk. |
|
SS-1.6.0 on Orion Rocky 9 w/ Any idea when the GSI will be updated to use this on Orion? |
|
One of the CRTM fix files is incorrect and needs to be updated on Orion. I have opened issue JCSDA/spack-stack#1158 to correct this. |
|
Thank you @DavidHuber-NOAA for reporting the Orion Rocky 9 spack-stack 1.6.0 crtm issue. |
|
@TingLei-NOAA , what is the status of your Orion Rocky 9 GSI tests? |
|
The CRTM-fix file has been corrected, so the GSI should be good to proceed. |
@emilyhcliu , after you built |
I notice @emilyhcliu is also updating to bufr 12. Is that a source of the slowdown? |
|
Also, I note that @emilyhcliu is pointing to the CentOS spack-stack build. It should be prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/gsi-addon-env-rocky9/install/modulefiles/Core") |
|
@aerorahul , thank you for noting this. I did not include @emilyhcliu 's I wonder if some of the Rocky 9 modules were built with debug options or lower levels of optimization. Alternatively, there could be environment variables we need to add, remove, or change to efficiently run |
|
Thank you @DavidHuber-NOAA . I checked my local copy of I am working in |
|
Orion whereas Hercules The CRTM coefficients in these two For example, the Orion and Hercules The Orion initial penalties Hercules initial penalties Something isn't right on Orion. |
|
Rerun Orion case using Hercules crtm coefficients. With this change the initial radiance penalty matches the hercules run. However, the Orion |
|
@RussTreadon-NOAA This is interesting/worrisome RE the CRTM fix files. Orion's > pwd
/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env-rocky9/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-qls55kd/fix
> for file in *; do
> cmp $file /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env/install/intel/2022.0.2/crtm-fix-2.4.0.1_emc-ezbyzji/fix/$file
> done
>
> for file in /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env/install/intel/2022.0.2/crtm-fix-2.4.0.1_emc-ezbyzji/fix/*; do
> f=$(basename $file)
> cmp $f $file
> done
cmp: amsua_metop-c.SpcCoeff.noACC.bin: No such file or directoryThe missing Performing the commands > for file in /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env/install/intel/2022.0.2/crtm-fix-2.4.0.1_emc-ezbyzji/fix/*; do
> f=$(basename $file)
> cmp $file /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-2os2hw2/fix/$f
> doneproduces a list of 445 files that differ. I'll report these findings to JCSDA. I will suggest that they compare the fix files on Orion and Hercules with those in production on WCOSS2 to verify which are correct. |
|
Build GSI Run ctests with following results These run times are noticeably higher than what was observed when Orion ran Centos-7. For example, PR #746 contains the following Orion ctest timings @KateFriedman-NOAA , @DavidHuber-NOAA , and @aerorahul : have any other g-w components reported increased wall times on Orion Rocky 9? |
|
@RussTreadon-NOAA The only other component that I know of that has run their tests is UFS_Utils (ufs-community/UFS_UTILS#966). No wall times had to be increased for that PR. |
|
Thank you @DavidHuber-NOAA for your reply. I don't know what to try next or who to contact. Does the spack-stack team run unit tests for their installations? It would be good to get confirmation that Orion Rocky-9 modules run as fast as their Centos-7 counterparts. |
|
Agreed. @AlexanderRichert-NOAA we are seeing significant slowdowns in the GSI on Orion after the OS upgrade to Rocky-9 and also significantly slower runtimes compared to Hercules. Do you know if tests were run on the spack-stack libraries and/or if they are not as optimized as they were under CentOS? |
|
Work for this issue will be done in RussTreadon-NOAA:feature/orion_rocky9 |
|
@TingLei-NOAA , have you built |
Learn that @TingLei-NOAA is on leave. |
|
Open ticket RDHPCS#2024062754000098 with Orion Helpdesk to report |
Received the following from RDHPCS Management
This issue is opened to document the updating of
modulefiles/gsi_orion.intel.luato Rocky 9The text was updated successfully, but these errors were encountered: