New Velocity Potential and Stream Function Calculations

[KarinaAsmar-NOAA]

This PR adds the CPC-requested streamfunction and velocity potential at 200mb to SFS. It is meant to resolve [Issue #902 ] and update on the runtime issues from the previous PR #951 . Job scripts used for testing are in WCOSS2: /lfs/h2/emc/vpppg/noscrub/karina.asmar/vpot_strm/UPP (submit_run_gfsv16_wcoss2.sh and submit_run_sfs_wcoss2.sh).

[KarinaAsmar-NOAA]

@WenMeng-NOAA We have updated the computation for streamfunction and velocity potential for compatibility with UPP parallelization. The subroutine now uses a poisson solver with a convergence tester to reduce runtimes. Comparison of spectral and numerical results is here

[GeorgeVandenberghe-NOAA]

The poisson solver works but is substantially slower than the gather --> stptranf --> scatter operation to solve the equation spectrally. The gather is much cheaper than it sounds because it is TWO fields, (U and V) at one level, not the several hundred such fields that comprise state plus derivative variables. A gather of two fields takes about 0.06 seconds on hera at GFS (high) resolution and it hard to even measure at CFS resolution. The relaxations take about 10 seconds on hera for chi and psi together and less than a second for the spectral solver. Spectral solver takes 27 seconds for the GFS (high) resolution case while the poisson solver takes 161 seconds. Spectral solver takes less than a second for the CFS case while the poisson solver takes about three seconds.

[WenMeng-NOAA]

@KarinaAsmar-NOAA Clean up the debugging code.

[KarinaAsmar-NOAA]

@GeorgeVandenberghe-NOAA Would you please clean up the debugging part of COLLECT_LOC.f? Let me know when done and I'll push it to this branch.

[WenMeng-NOAA]

@KarinaAsmar-NOAA Clean up the debugging code in this routine.

[WenMeng-NOAA]

@KarinaAsmar-NOAA Please update fortran code in the subroutine as following format:

• all lowercase
• indentation for loops, conditionals and blocks

[WenMeng-NOAA]

@DusanJovic-NOAA @junwang-noaa For your reference, I ran UFS-WM RT 'cpld_control_sfs_intel' on Hera:

1. develop branch,
   run directory: /home/Wen.Meng/stmp2/FV3_RT/rt_449231/cpld_control_sfs_intel
   inline post runtime: 5 seconds

2. develop branch, replace UPP version #PR 1072, include velocity potential and stream function generation
   run directory: /home/Wen.Meng/stmp2/FV3_RT/rt_700773/cpld_control_sfs_intel
   inline post runtime: 8.7 seconds

[GeorgeVandenberghe-NOAA]

The allreduce is numerically critical. What it is doing is checking for the maximum difference between the previous value of psi/chi and the result at the end of the next iteration. If the allreduce is left out, this difference is only evaluated on the subdomain associated with MPI rank 0. The allreduce does a max operation on this error on ALL of the ranks. Removing the allreduce will result in faster exit since convergence is only checked on rank 0 and a "bad" rank's larger errors will not be removed by further iterations. The ROOT problem with this whole poisson solver is the extremely slow convergence for all methods tried to date. Note, if we do get rid of the allreduce, rank 0 is one of the best ranks to test on because it's on a corner domain and likely where convergence IS slowest.

[KarinaAsmar-NOAA]

The allreduce is numerically critical. What it is doing is checking for the maximum difference between the previous value of psi/chi and the result at the end of the next iteration. If the allreduce is left out, this difference is only evaluated on the subdomain associated with MPI rank 0. The allreduce does a max operation on this error on ALL of the ranks. Removing the allreduce will result in faster exit since convergence is only checked on rank 0 and a "bad" rank's larger errors will not be removed by further iterations. The ROOT problem with this whole poisson solver is the extremely slow convergence for all methods tried to date. Note, if we do get rid of the allreduce, rank 0 is one of the best ranks to test on because it's on a corner domain and likely where convergence IS slowest.

@GeorgeVandenberghe-NOAA It seems that the mpi_allreduce call causes the inline post runs to crash (@WenMeng-NOAA , plese confirm if the inline keeps crashing after removal of allreduce). Removing it does not seem to affect the SFS results (see pptx).

[GeorgeVandenberghe-NOAA]

Crash with allreduce of a single word variable on all tasks just shouldn't happen. If it does, there is something else wrong, perhaps a fundamental error in the specified MPI communicator in allreduce or a problem (again fundamental) in one of the other arguments. a real8 variable with mpi_real specified as the type is one possibility I will check. BUt we're moving to a fixed number of iterations anyway and not testing for convergence so then allreduce is not needed.

…

On Wed, Dec 18, 2024 at 10:25 AM KarinaAsmar-NOAA ***@***.***> wrote: The allreduce is numerically critical. What it is doing is checking for the maximum difference between the previous value of psi/chi and the result at the end of the next iteration. If the allreduce is left out, this difference is only evaluated on the subdomain associated with MPI rank 0. The allreduce does a max operation on this error on ALL of the ranks. Removing the allreduce will result in faster exit since convergence is only checked on rank 0 and a "bad" rank's larger errors will not be removed by further iterations. The ROOT problem with this whole poisson solver is the extremely slow convergence for all methods tried to date. Note, if we do get rid of the allreduce, rank 0 is one of the best ranks to test on because it's on a corner domain and likely where convergence IS slowest. @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> It seems that the mpi_allreduce call causes the inline post runs to crash ( @WenMeng-NOAA <https://github.com/WenMeng-NOAA> , plese confirm if the inline keeps crashing after removal of allreduce). Removing it does not seem to affect the SFS results (see pptx <#1072 (comment)>). — Reply to this email directly, view it on GitHub <#1072 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FVEJGJZI4GU7IQWKW32GGHYDAVCNFSM6AAAAABQK7FXP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJRGYYTEMBWGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[KarinaAsmar-NOAA]

@WenMeng-NOAA Would you please run your tests with the latest changes? Let me know if anything fails and/or if runtimes are not acceptable.

@KarinaAsmar-NOAA Your latest commit removed mpi_allreduce which significantly improved the runtime of the offline post. May I know the reason of this change?

@WenMeng-NOAA I removed it thinking that could be the issue that is causing the inline post to crash, since it is a collective operation requiring the reduced values to be called and distributed across all processes. We were using it to help the poisson loops converge. After removing it, the addition of the openMP directives and the reduced number of loops helped keep accuracy of results and reduce runtimes. You can see the SFS result comparison STRM_VPOT_Poisson_Results_v5.pptx. I haven't been able to test GFS on Hera with the spectral method, but once WCOSS2 is back up I can generate some results. If the inline post works for this test, then it was probably the mpi_allreduce that was causing the runtime/memory issue.

Latest comparisons of SFS and GFS with 100,000 iterations and no 'mpi_allreduce' here:
STRM_VPOT_Poisson_Results_v5.pptx

[WenMeng-NOAA]

@KarinaAsmar-NOAA @JesseMeng-NOAA What is the 'pval' used for? I don't see it initialized or calculated?

[JesseMeng-NOAA]

should be
edif=chi(i,j)-ptmp(i,j)

[KarinaAsmar-NOAA]

@WenMeng-NOAA The 'pval' is for evaluating errors across iterations. I missed adding it when restoring the allreduce, it is added now.

[WenMeng-NOAA]

Here are the runtime tests for the offline post:

GFS(C768) 120 tasks || develop: 142 seconds  ||  PR #1072: 188 seconds
GEFS (C384) 48 tasks || develop: 52 seconds   || PR #1072: 77 seconds
SFS (C96) 48 tasks  ||  develop: 3.86 seconds   || PR #1072: 5.4 seconds

[WenMeng-NOAA]

Here are the runtime test for the inline post with the UFS RT 'cpld_control_sfs' on Hera:

Without  velocity potential and stream function generation: 5 seconds
   run directory: /scratch1/NCEPDEV/stmp2/Wen.Meng/FV3_RT/rt_449231-develop/cpld_control_sfs_intel
With velocity potential and stream function generation: 9 seconds
   run directory: /scratch1/NCEPDEV/stmp2/Wen.Meng/FV3_RT/rt_2703606-sfs/cpld_control_sfs_intel

@DusanJovic-NOAA and @junwang-noaa Please let me know your comments on this PR.

[WenMeng-NOAA]

@KarinaAsmar-NOAA Comment out the debug code from line 4967 to 4969.

[WenMeng-NOAA]

@KarinaAsmar-NOAA Comment out this line.

[WenMeng-NOAA]

@KarinaAsmar-NOAA Comment out this line.