Add bufr2ioda ctest #207

SamuelDegelia-NOAA · 2024-10-18T01:52:48Z

This is a draft PR to add a bufr2ioda ctest following #171. This shows how we can use the iodaconv_comp.sh tool to compare against a reference IODA file.

Here, a new ctest is added called rrfs_bufr2ioda_msonet. It runs bufr2ioda.x for a given prepbufr file. Then, iodaconv_comp_rrfs.sh is run to compare the output between the ioda_msonet.nc generated during the ctest and a reference version.

Output from ctest:

1/1 Test #7: rrfs_bufr2ioda_msonet ............   Passed  585.70 sec

The following tests passed:
        rrfs_bufr2ioda_msonet

100% tests passed, 0 tests failed out of 1

Label Time Summary:
mpi            = 585.70 sec*proc (1 test)
rdas-bundle    = 585.70 sec*proc (1 test)
script         = 585.70 sec*proc (1 test)

Total Test time (real) = 585.71 sec

Output from ctest but changing the pressure scaling from 100 to 90 in the input yaml file:

7: DIFFER : VARIABLE : pressure : POSITION : [521497] : VALUES : 101930 <> 91737 : PERCENT : 10
...
...
7: DIFFER : VARIABLE : pressure : POSITION : [521504] : VALUES : 101930 <> 91737 : PERCENT : 10
1/1 Test #7: rrfs_bufr2ioda_msonet ............***Failed   38.01 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary:
mpi            =  38.01 sec*proc (1 test)
rdas-bundle    =  38.01 sec*proc (1 test)
script         =  38.01 sec*proc (1 test)

Total Test time (real) =  38.10 sec

The following tests FAILED:
          7 - rrfs_bufr2ioda_msonet (Failed)

There is still some work to clean up the additions to rrfs-test/CMakeLists.txt, and I would like to add a test for the python bufr2ioda converter. But this shows that this type of test is possible and can be added to RDASApp pretty easily.

SamuelDegelia-NOAA · 2024-10-21T16:52:35Z

I think this PR is ready for review now. Since PR #192 is still under review, we can plan to add a ctest later for the python-based converter.

(eva) [Samuel.Degelia@hfe10 rrfs-test]$ (eva) [Samuel.Degelia@hfe10 rrfs-test]$ ctest -j7
Test project /scratch1/BMC/zrtrr/Samuel.Degelia/RDASApp_dev/RDASApp/build/rrfs-test
    Start 3: rrfs_mpasjedi_2024052700_Ens3Dvar
    Start 4: rrfs_mpasjedi_2024052700_getkf_observer
    Start 6: rrfs_mpasjedi_2024052700_bumploc
    Start 1: rrfs_fv3jedi_hyb_2022052619
    Start 2: rrfs_fv3jedi_letkf_2022052619
    Start 7: rrfs_bufr2ioda_msonet
1/7 Test #7: rrfs_bufr2ioda_msonet .....................   Passed   42.52 sec
2/7 Test #2: rrfs_fv3jedi_letkf_2022052619 .............   Passed   57.09 sec
3/7 Test #1: rrfs_fv3jedi_hyb_2022052619 ...............   Passed  118.83 sec
4/7 Test #4: rrfs_mpasjedi_2024052700_getkf_observer ...   Passed  1943.64 sec
    Start 5: rrfs_mpasjedi_2024052700_getkf_solver
5/7 Test #6: rrfs_mpasjedi_2024052700_bumploc ..........   Passed  2083.63 sec
6/7 Test #3: rrfs_mpasjedi_2024052700_Ens3Dvar .........   Passed  4477.88 sec
7/7 Test #5: rrfs_mpasjedi_2024052700_getkf_solver .....   Passed  3735.32 sec

100% tests passed, 0 tests failed out of 7

Label Time Summary:
mpi            = 12458.90 sec*proc (7 tests)
rdas-bundle    = 12458.90 sec*proc (7 tests)
script         = 12458.90 sec*proc (7 tests)

Total Test time (real) = 5679.05 sec

delippi · 2024-10-22T15:21:00Z

@SamuelDegelia-NOAA, thanks for putting this together. Should we wait until I finalize the mesonet bufr2ioda yaml in my other PR #183 and Praveen's PR #181? I was also waiting to finalize those with my other PR #188 to make sure I didn't miss anything. It might not matter since we can always just update accordingly and we will just have everything already in place.

SamuelDegelia-NOAA · 2024-10-22T15:22:53Z

@delippi That is probably a good idea. Works for me!

rrfsbot · 2024-10-22T23:27:41Z

FAILED on jet

started build_and_test on jet at UTC time: Tue Oct 22 22:21:25 UTC 2024
finished at UTC time: Tue Oct 22 23:27:40 UTC 2024

Test project /lfs5/BMC/wrfruc/rrfsbot/PRs_RDASApp/207/build/rrfs-test
    Start 4: rrfs_mpasjedi_2024052700_getkf_observer
    Start 1: rrfs_fv3jedi_hyb_2022052619
    Start 2: rrfs_fv3jedi_letkf_2022052619
    Start 3: rrfs_mpasjedi_2024052700_Ens3Dvar
    Start 6: rrfs_mpasjedi_2024052700_bumploc
    Start 7: rrfs_bufr2ioda_msonet
1/7 Test #2: rrfs_fv3jedi_letkf_2022052619 .............   Passed   92.27 sec
2/7 Test #1: rrfs_fv3jedi_hyb_2022052619 ...............   Passed  157.97 sec
3/7 Test #6: rrfs_mpasjedi_2024052700_bumploc ..........   Passed  222.47 sec
4/7 Test #7: rrfs_bufr2ioda_msonet .....................***Failed  357.25 sec
5/7 Test #4: rrfs_mpasjedi_2024052700_getkf_observer ...   Passed  468.27 sec
    Start 5: rrfs_mpasjedi_2024052700_getkf_solver
6/7 Test #3: rrfs_mpasjedi_2024052700_Ens3Dvar .........   Passed  658.26 sec
7/7 Test #5: rrfs_mpasjedi_2024052700_getkf_solver .....   Passed  1626.46 sec

86% tests passed, 1 tests failed out of 7

Label Time Summary:
mpi            = 3582.94 sec*proc (7 tests)
rdas-bundle    = 3582.94 sec*proc (7 tests)
script         = 3582.94 sec*proc (7 tests)

Total Test time (real) = 2094.77 sec

The following tests FAILED:
	  7 - rrfs_bufr2ioda_msonet (Failed)
Errors while running CTest
Output from these tests are in: /lfs5/BMC/wrfruc/rrfsbot/PRs_RDASApp/207/build/rrfs-test/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

workdir: /lfs5/BMC/wrfruc/rrfsbot/PRs_RDASApp/207

rrfsbot · 2024-10-22T23:31:03Z

PASSED on hercules

started build_and_test on hercules at UTC time: Tue Oct 22 22:21:10 UTC 2024
finished at UTC time: Tue Oct 22 23:31:03 UTC 2024

Test project /work/noaa/wrfruc/rrfsbot/PRs_RDASApp/207/build/rrfs-test
    Start 4: rrfs_mpasjedi_2024052700_getkf_observer
    Start 1: rrfs_fv3jedi_hyb_2022052619
    Start 2: rrfs_fv3jedi_letkf_2022052619
    Start 3: rrfs_mpasjedi_2024052700_Ens3Dvar
    Start 6: rrfs_mpasjedi_2024052700_bumploc
    Start 7: rrfs_bufr2ioda_msonet
1/7 Test #7: rrfs_bufr2ioda_msonet .....................   Passed   48.75 sec
2/7 Test #1: rrfs_fv3jedi_hyb_2022052619 ...............   Passed  148.69 sec
3/7 Test #2: rrfs_fv3jedi_letkf_2022052619 .............   Passed  422.04 sec
4/7 Test #6: rrfs_mpasjedi_2024052700_bumploc ..........   Passed  511.34 sec
5/7 Test #4: rrfs_mpasjedi_2024052700_getkf_observer ...   Passed  1078.60 sec
    Start 5: rrfs_mpasjedi_2024052700_getkf_solver
6/7 Test #3: rrfs_mpasjedi_2024052700_Ens3Dvar .........   Passed  1481.02 sec
7/7 Test #5: rrfs_mpasjedi_2024052700_getkf_solver .....   Passed  1259.41 sec

100% tests passed, 0 tests failed out of 7

Label Time Summary:
mpi            = 4949.84 sec*proc (7 tests)
rdas-bundle    = 4949.84 sec*proc (7 tests)
script         = 4949.84 sec*proc (7 tests)

Total Test time (real) = 2338.02 sec

workdir: /work/noaa/wrfruc/rrfsbot/PRs_RDASApp/207

SamuelDegelia-NOAA · 2024-10-23T04:14:19Z

The failed ctest on Jet was due to the job running out of memory. I switched from using --exclusive to --mem=64G. The ctest is working for me now, but feel free to rerun the CI @guoqing-noaa

guoqing-noaa · 2024-10-23T04:16:45Z

The failed ctest on Jet was due to the job running out of memory. I switched from using --exclusive to --mem=64G. The ctest is working for me now, but feel free to rerun the CI @guoqing-noaa

@SamuelDegelia-NOAA Thanks for fixing this!

rrfsbot · 2024-10-23T05:21:25Z

PASSED on hera

started build_and_test on hera at UTC time: Wed Oct 23 04:23:18 UTC 2024
finished at UTC time: Wed Oct 23 05:21:24 UTC 2024

Test project /scratch1/NCEPDEV/fv3-cam/rrfsbot/PRs_RDASApp/207/build/rrfs-test
    Start 4: rrfs_mpasjedi_2024052700_getkf_observer
    Start 1: rrfs_fv3jedi_hyb_2022052619
    Start 2: rrfs_fv3jedi_letkf_2022052619
    Start 3: rrfs_mpasjedi_2024052700_Ens3Dvar
    Start 6: rrfs_mpasjedi_2024052700_bumploc
    Start 7: rrfs_bufr2ioda_msonet
1/7 Test #7: rrfs_bufr2ioda_msonet .....................   Passed   27.44 sec
2/7 Test #2: rrfs_fv3jedi_letkf_2022052619 .............   Passed   37.08 sec
3/7 Test #1: rrfs_fv3jedi_hyb_2022052619 ...............   Passed   98.42 sec
4/7 Test #6: rrfs_mpasjedi_2024052700_bumploc ..........   Passed  118.85 sec
5/7 Test #4: rrfs_mpasjedi_2024052700_getkf_observer ...   Passed  385.46 sec
    Start 5: rrfs_mpasjedi_2024052700_getkf_solver
6/7 Test #3: rrfs_mpasjedi_2024052700_Ens3Dvar .........   Passed  419.28 sec
7/7 Test #5: rrfs_mpasjedi_2024052700_getkf_solver .....   Passed  1255.62 sec

100% tests passed, 0 tests failed out of 7

Label Time Summary:
mpi            = 2342.14 sec*proc (7 tests)
rdas-bundle    = 2342.14 sec*proc (7 tests)
script         = 2342.14 sec*proc (7 tests)

Total Test time (real) = 1641.43 sec

workdir: /scratch1/NCEPDEV/fv3-cam/rrfsbot/PRs_RDASApp/207

rrfsbot · 2024-10-23T05:32:12Z

PASSED on jet

started build_and_test on jet at UTC time: Wed Oct 23 04:21:51 UTC 2024
finished at UTC time: Wed Oct 23 05:32:11 UTC 2024

Test project /lfs5/BMC/wrfruc/rrfsbot/PRs_RDASApp/207/build/rrfs-test
    Start 4: rrfs_mpasjedi_2024052700_getkf_observer
    Start 1: rrfs_fv3jedi_hyb_2022052619
    Start 2: rrfs_fv3jedi_letkf_2022052619
    Start 3: rrfs_mpasjedi_2024052700_Ens3Dvar
    Start 6: rrfs_mpasjedi_2024052700_bumploc
    Start 7: rrfs_bufr2ioda_msonet
1/7 Test #2: rrfs_fv3jedi_letkf_2022052619 .............   Passed   79.67 sec
2/7 Test #1: rrfs_fv3jedi_hyb_2022052619 ...............   Passed  145.59 sec
3/7 Test #7: rrfs_bufr2ioda_msonet .....................   Passed  166.68 sec
4/7 Test #6: rrfs_mpasjedi_2024052700_bumploc ..........   Passed  213.98 sec
5/7 Test #4: rrfs_mpasjedi_2024052700_getkf_observer ...   Passed  474.40 sec
    Start 5: rrfs_mpasjedi_2024052700_getkf_solver
6/7 Test #3: rrfs_mpasjedi_2024052700_Ens3Dvar .........   Passed  646.48 sec
7/7 Test #5: rrfs_mpasjedi_2024052700_getkf_solver .....   Passed  1836.51 sec

100% tests passed, 0 tests failed out of 7

Label Time Summary:
mpi            = 3563.32 sec*proc (7 tests)
rdas-bundle    = 3563.32 sec*proc (7 tests)
script         = 3563.32 sec*proc (7 tests)

Total Test time (real) = 2310.95 sec

workdir: /lfs5/BMC/wrfruc/rrfsbot/PRs_RDASApp/207

guoqing-noaa · 2024-10-23T07:36:55Z

FYI, some of Hercules nodes are in maintenance and that's why we have not gotten the CI test results from Hercules yet.

guoqing-noaa · 2024-10-23T20:09:30Z

Let's merge this first to facilitate PR #209

guoqing-noaa

LGTM.

SamuelDegelia-NOAA added 2 commits October 18, 2024 01:47

First working example

5422ae7

Remove duplicate line

348c254

SamuelDegelia-NOAA mentioned this pull request Oct 18, 2024

create bufr to ioda ctest #171

Open

Clean up some of CMake and increase parallel runs of ctest from 6 to 7

f074500

Reorganize a bit

76eeb42

SamuelDegelia-NOAA marked this pull request as ready for review October 21, 2024 16:53

SamuelDegelia-NOAA requested review from ShunLiu-NOAA, delippi, guoqing-noaa and TingLei-NOAA October 21, 2024 16:53

TingLei-NOAA previously approved these changes Oct 22, 2024

View reviewed changes

hu5970 self-requested a review October 22, 2024 21:17

hu5970 previously approved these changes Oct 22, 2024

View reviewed changes

Merge branch 'develop' into feature/bufr2ioda_ctest

9be3a90

guoqing-noaa added test_hercules test_jet labels Oct 22, 2024

rrfsbot added running_hercules running_jet Failed_jet and removed test_hercules test_jet running_jet labels Oct 22, 2024

rrfsbot added Passed_hercules and removed running_hercules labels Oct 22, 2024

Change exclusive to memory request

2b6526e

SamuelDegelia-NOAA dismissed stale reviews from TingLei-NOAA and hu5970 via 2b6526e October 23, 2024 04:12

guoqing-noaa added test_hera test_jet and removed Failed_jet labels Oct 23, 2024

rrfsbot added running_hera running_jet Passed_hera and removed test_hera test_jet running_hera labels Oct 23, 2024

guoqing-noaa added test_hercules and removed Passed_hercules labels Oct 23, 2024

rrfsbot added running_hercules Passed_jet and removed test_hercules running_jet labels Oct 23, 2024

SamuelDegelia-NOAA mentioned this pull request Oct 23, 2024

Update OOPS and use reduce obs space filter action for pre-filters #209

Closed

guoqing-noaa approved these changes Oct 23, 2024

View reviewed changes

guoqing-noaa merged commit a5e1c2f into NOAA-EMC:develop Oct 23, 2024
1 check passed

SamuelDegelia-NOAA deleted the feature/bufr2ioda_ctest branch October 23, 2024 23:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bufr2ioda ctest #207

Add bufr2ioda ctest #207

SamuelDegelia-NOAA commented Oct 18, 2024

SamuelDegelia-NOAA commented Oct 21, 2024

delippi commented Oct 22, 2024

SamuelDegelia-NOAA commented Oct 22, 2024

rrfsbot commented Oct 22, 2024

rrfsbot commented Oct 22, 2024

SamuelDegelia-NOAA commented Oct 23, 2024

guoqing-noaa commented Oct 23, 2024

rrfsbot commented Oct 23, 2024

rrfsbot commented Oct 23, 2024

guoqing-noaa commented Oct 23, 2024

guoqing-noaa commented Oct 23, 2024

guoqing-noaa left a comment

Add bufr2ioda ctest #207

Add bufr2ioda ctest #207

Conversation

SamuelDegelia-NOAA commented Oct 18, 2024

SamuelDegelia-NOAA commented Oct 21, 2024

delippi commented Oct 22, 2024

SamuelDegelia-NOAA commented Oct 22, 2024

rrfsbot commented Oct 22, 2024

rrfsbot commented Oct 22, 2024

SamuelDegelia-NOAA commented Oct 23, 2024

guoqing-noaa commented Oct 23, 2024

rrfsbot commented Oct 23, 2024

rrfsbot commented Oct 23, 2024

guoqing-noaa commented Oct 23, 2024

guoqing-noaa commented Oct 23, 2024

guoqing-noaa left a comment

Choose a reason for hiding this comment