Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add capability to use RAP/HRRR MASSDEN as smoke IC/LBCs #923

Merged
merged 18 commits into from
Nov 12, 2024

Conversation

jordanschnell
Copy link
Contributor

@jordanschnell jordanschnell commented Mar 20, 2024

DESCRIPTION OF CHANGES:

Previously, massden/smoke was not part of the variable list to ingest into IC/LBCs from RAP/HRRR models. This PR adds the massden/smoke to the mapping table and adds the necessary functionality to read in and convert to the correct units. An additional allocated array is created when using RAP/HRRR for ICs/LBCs during the smoke read to be able to convert units. Other models (e.g., GFS) will not attempt to read in smoke.

TESTS CONDUCTED:

  • Compile branch on all Tier 1 machines using Intel (Orion, Jet, Hera, Hercules and WCOSS2). Done using d126cb6.
  • Compile branch on Hera using GNU. Done using d126cb6.
  • Compile branch in 'Debug' mode on WCOSS2 with no warnings. Done using d126cb6.
  • Run unit tests locally on any Tier 1 machine. Done on Hera using d126cb6 (and Gnu). All tests passed.
  • Run chgres_cube consistency tests locally on all Tier 1 machines. Done using d126cb6. All tests - and the new test - passed.

Additional tests performed.

DEPENDENCIES:

none

DOCUMENTATION:

Update readthedocs: https://schnel.readthedocs.io/en/rap_icbc/ufs_utils.html#variable-mapping-varmap-table

ISSUE:

Fixes #851.

CONTRIBUTORS:

Johana Romero-Alvarez (CIRES/NOAA GSL)

Screenshot 2024-03-1 <img width="725" alt="Screenshot 2024-03-19 at 6 37 28 PM" src="https://github.com/ufs-community/UFS_UTILS/assets/46031459/2fd276fa-4e8a-481f-9ed2-34c7a35eda50"> 9 at 6 35 55 PM
Screenshot 2024-03-19 at 6 39 17 PM

Screenshot 2024-03-19 at 6 37 28 PM

@GeorgeGayno-NOAA
Copy link
Collaborator

I am curious what project or implementation this PR supports.

@jordanschnell
Copy link
Contributor Author

I am curious what project or implementation this PR supports.

I'm not sure exactly which details you need, but this is largely research oriented e.g., For computational efficiency, for retrospective experiments prior to RRFS-A SD implementation, we often only want to run the CONUS domain but would still like e.g., Canadian originated smoke to enter the domain (e.g., DRSA FIRE3). The same could be said for using HRRR IC/BCs for the fire nest. This wIll also support NOAA ERB funded work.

@GeorgeGayno-NOAA
Copy link
Collaborator

We ask that developers run the consistency tests located in ./reg_tests/chgres_cube. I ran them on Hera (CentOS nodes). Two of the tests failed. (The number of tracers in the gfs_ctrl.nc file differ from the baseline). Can you run these tests on Jet? Just invoke ./driver.jet.sh.

@jordanschnell
Copy link
Contributor Author

We ask that developers run the consistency tests located in ./reg_tests/chgres_cube. I ran them on Hera (CentOS nodes). Two of the tests failed. (The number of tracers in the gfs_ctrl.nc file differ from the baseline). Can you run these tests on Jet? Just invoke ./driver.jet.sh.

Will do - and will look into these failures. Thank you for the information.

@jordanschnell
Copy link
Contributor Author

jordanschnell commented Mar 20, 2024

We ask that developers run the consistency tests located in ./reg_tests/chgres_cube. I ran them on Hera (CentOS nodes). Two of the tests failed. (The number of tracers in the gfs_ctrl.nc file differ from the baseline). Can you run these tests on Jet? Just invoke ./driver.jet.sh.

I ran the tests on Jet and also received the two failures (13-km CONUS RAP W/ GSD PHYSICS AND SFC FROM FILE GRIB2 and 3-km CONUS HRRR W/ GSD PHYSICS AND SFC FROM FILE GRIB2). This makes sense though, since smoke is being added to the files and so e.g., there is an additional tracer in the out files. How would you like this to be addressed?

@GeorgeGayno-NOAA
Copy link
Collaborator

We ask that developers run the consistency tests located in ./reg_tests/chgres_cube. I ran them on Hera (CentOS nodes). Two of the tests failed. (The number of tracers in the gfs_ctrl.nc file differ from the baseline). Can you run these tests on Jet? Just invoke ./driver.jet.sh.

I ran the tests on Jet and also received the two failures (13-km CONUS RAP W/ GSD PHYSICS AND SFC FROM FILE GRIB2 and 3-km CONUS HRRR W/ GSD PHYSICS AND SFC FROM FILE GRIB2). This makes sense though, since smoke is being added to the files and so e.g., there is an additional tracer in the out files. How would you like this to be addressed?

The number of tracers in gfs_ctrl.nc should match those the in the atmospheric 'out' file.

@GeorgeGayno-NOAA
Copy link
Collaborator

As a test, I ran the 13km.conus.rap.grib2 regression test with all the smoke records removed from the input GRIB2 file.

The test script was updated as follows:

--- a/reg_tests/chgres_cube/13km.conus.rap.grib2.sh
+++ b/reg_tests/chgres_cube/13km.conus.rap.grib2.sh
@@ -16,9 +16,9 @@ export CRES=778
 export KMRES=13km
 export FIXfv3=${HOMEreg}/fix/RRFS_CONUS_${KMRES}

-export COMIN=${HOMEreg}/input_data/rap.grib2
+export COMIN=/work2/noaa/da/ggayno/save/UFS_UTILS.schnell/data

-export GRIB2_FILE_INPUT=1921221000900
+export GRIB2_FILE_INPUT=rap.grb2
 export VCOORD_FILE=${HOMEufs}/fix/am/global_hyblev.l64.txt
 export VARMAP_FILE=${HOMEufs}/parm/varmap_tables/GSDphys_var_map.txt

The modified input data and the script used to create it is here: /work2/noaa/da/ggayno/save/UFS_UTILS.schnell/data

@jordanschnell
Copy link
Contributor Author

We ask that developers run the consistency tests located in ./reg_tests/chgres_cube. I ran them on Hera (CentOS nodes). Two of the tests failed. (The number of tracers in the gfs_ctrl.nc file differ from the baseline). Can you run these tests on Jet? Just invoke ./driver.jet.sh.

I ran the tests on Jet and also received the two failures (13-km CONUS RAP W/ GSD PHYSICS AND SFC FROM FILE GRIB2 and 3-km CONUS HRRR W/ GSD PHYSICS AND SFC FROM FILE GRIB2). This makes sense though, since smoke is being added to the files and so e.g., there is an additional tracer in the out files. How would you like this to be addressed?

The number of tracers in gfs_ctrl.nc should match those the in the atmospheric 'out' file.

But if the end result of my modifications is to add a new tracer to the out file, won't it automatically fail?

@GeorgeGayno-NOAA
Copy link
Collaborator

For the first test using the modified input data, I updated the varmap file as follows:

--- a/parm/varmap_tables/GSDphys_var_map.txt
+++ b/parm/varmap_tables/GSDphys_var_map.txt
@@ -25,4 +25,4 @@ icetk               icetk     set_to_fill   265.0     S
 weasd               weasd     set_to_fill   0.0       S
 snod                snod      set_to_fill   0.0       S
 tprcp               tprcp     set_to_fill   0.0       S
-massden             smoke     set_to_fill   1E-12     T
+massden             smoke     stop          1E-12     T

Since there are no smoke records, processing should stop. And that is what the log file shows:

- CALL FieldGather TEMPERATURE.
 - READ PRESSURE FOR SMOKE CONVERSION.
 - FATAL ERROR:
 READING massden at level      1 hybrid       . TO MAKE THIS NON-FATAL, CHANGE S
 TOP TO SKIP FOR THIS VARIABLE IN YOUR VARMAP FILE.
 - IOSTAT IS:            0
Abort(999) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 0

@jordanschnell
Copy link
Contributor Author

As a test, I ran the 13km.conus.rap.grib2 regression test with all the smoke records removed from the input GRIB2 file.

The test script was updated as follows:

--- a/reg_tests/chgres_cube/13km.conus.rap.grib2.sh
+++ b/reg_tests/chgres_cube/13km.conus.rap.grib2.sh
@@ -16,9 +16,9 @@ export CRES=778
 export KMRES=13km
 export FIXfv3=${HOMEreg}/fix/RRFS_CONUS_${KMRES}

-export COMIN=${HOMEreg}/input_data/rap.grib2
+export COMIN=/work2/noaa/da/ggayno/save/UFS_UTILS.schnell/data

-export GRIB2_FILE_INPUT=1921221000900
+export GRIB2_FILE_INPUT=rap.grb2
 export VCOORD_FILE=${HOMEufs}/fix/am/global_hyblev.l64.txt
 export VARMAP_FILE=${HOMEufs}/parm/varmap_tables/GSDphys_var_map.txt

The modified input data and the script used to create it is here: /work2/noaa/da/ggayno/save/UFS_UTILS.schnell/data

I don't have access to this system. Are you suggesting I add something to the code? Or the reg test?

@jordanschnell
Copy link
Contributor Author

For the first test using the modified input data, I updated the varmap file as follows:

--- a/parm/varmap_tables/GSDphys_var_map.txt
+++ b/parm/varmap_tables/GSDphys_var_map.txt
@@ -25,4 +25,4 @@ icetk               icetk     set_to_fill   265.0     S
 weasd               weasd     set_to_fill   0.0       S
 snod                snod      set_to_fill   0.0       S
 tprcp               tprcp     set_to_fill   0.0       S
-massden             smoke     set_to_fill   1E-12     T
+massden             smoke     stop          1E-12     T

Since there are no smoke records, processing should stop. And that is what the log file shows:

- CALL FieldGather TEMPERATURE.
 - READ PRESSURE FOR SMOKE CONVERSION.
 - FATAL ERROR:
 READING massden at level      1 hybrid       . TO MAKE THIS NON-FATAL, CHANGE S
 TOP TO SKIP FOR THIS VARIABLE IN YOUR VARMAP FILE.
 - IOSTAT IS:            0
Abort(999) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 0

Can you explain why you modified this?

@GeorgeGayno-NOAA
Copy link
Collaborator

For the first test using the modified input data, I updated the varmap file as follows:

--- a/parm/varmap_tables/GSDphys_var_map.txt
+++ b/parm/varmap_tables/GSDphys_var_map.txt
@@ -25,4 +25,4 @@ icetk               icetk     set_to_fill   265.0     S
 weasd               weasd     set_to_fill   0.0       S
 snod                snod      set_to_fill   0.0       S
 tprcp               tprcp     set_to_fill   0.0       S
-massden             smoke     set_to_fill   1E-12     T
+massden             smoke     stop          1E-12     T

Since there are no smoke records, processing should stop. And that is what the log file shows:

- CALL FieldGather TEMPERATURE.
 - READ PRESSURE FOR SMOKE CONVERSION.
 - FATAL ERROR:
 READING massden at level      1 hybrid       . TO MAKE THIS NON-FATAL, CHANGE S
 TOP TO SKIP FOR THIS VARIABLE IN YOUR VARMAP FILE.
 - IOSTAT IS:            0
Abort(999) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 0

Can you explain why you modified this?

The varmap files have several choices with how to process fields:

For the first test using the modified input data, I updated the varmap file as follows:

--- a/parm/varmap_tables/GSDphys_var_map.txt
+++ b/parm/varmap_tables/GSDphys_var_map.txt
@@ -25,4 +25,4 @@ icetk               icetk     set_to_fill   265.0     S
 weasd               weasd     set_to_fill   0.0       S
 snod                snod      set_to_fill   0.0       S
 tprcp               tprcp     set_to_fill   0.0       S
-massden             smoke     set_to_fill   1E-12     T
+massden             smoke     stop          1E-12     T

Since there are no smoke records, processing should stop. And that is what the log file shows:

- CALL FieldGather TEMPERATURE.
 - READ PRESSURE FOR SMOKE CONVERSION.
 - FATAL ERROR:
 READING massden at level      1 hybrid       . TO MAKE THIS NON-FATAL, CHANGE S
 TOP TO SKIP FOR THIS VARIABLE IN YOUR VARMAP FILE.
 - IOSTAT IS:            0
Abort(999) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 0

Can you explain why you modified this?

According to the README file, users can adjust how fields are processed using the varmap file. (See the instructions for column 3). Options include 'stop', 'skip' 'set_to_fill' etc. These different options need to be tested. So, I am temporarily modifying the varmap file.

https://github.com/ufs-community/UFS_UTILS/blob/develop/parm/varmap_tables/README

@GeorgeGayno-NOAA
Copy link
Collaborator

Next, test the 'skip' function. Modify the varmap table as follows:

--- a/parm/varmap_tables/GSDphys_var_map.txt
+++ b/parm/varmap_tables/GSDphys_var_map.txt
@@ -25,4 +25,4 @@ icetk               icetk     set_to_fill   265.0     S
 weasd               weasd     set_to_fill   0.0       S
 snod                snod      set_to_fill   0.0       S
 tprcp               tprcp     set_to_fill   0.0       S
-massden             smoke     set_to_fill   1E-12     T
+massden             smoke     skip          1E-12     T

The log file shows:

WARNING: SKIPPING massden IN FILE
WARNING: SKIPPING massden IN FILE
WARNING: SKIPPING massden IN FILE

And the out.atm.tile7.nc file contains a smoke record. Because the input GRIB2 file does not contain smoke, that was unexpected. According to the README, when 'skip' is used, there should not be a smoke record.

Herc /work2/noaa/stmp/ggayno/reg-tests/chgres-cube.skip/13km_conus_rap_grib2\ $ ncdump -h out.atm.tile7.nc
        float ice_aero(lev, lat, lon) ;
                ice_aero:coordinates = "geolon geolat" ;
        float sgs_tke(lev, lat, lon) ;
                sgs_tke:coordinates = "geolon geolat" ;
        float smoke(lev, lat, lon) ;
                smoke:coordinates = "geolon geolat" ;
        float u_w(lev, lat, lonp) ;
                u_w:coordinates = "geolon_w geolat_w" ;

@LarissaReames-NOAA and @JeffBeck-NOAA - you understand the GRIB2 option better I do. What is the expected result in this case?

@GeorgeGayno-NOAA
Copy link
Collaborator

@jordanschnell - this PR has not been updated in a while. There is more work required if you want to merge it. What do you want to do?

@jordanschnell
Copy link
Contributor Author

@GeorgeGayno-NOAA - We would like to merge it, however, I was not sure what else is required. Your question to Jeff and Larissa went unanswered and I am unsure of what the issue is. It seems like there may be other issues though (#851)

@LarissaReames-NOAA
Copy link
Collaborator

I think what @jordanschnell needs to do is create his own var mapping file based on the GSD var mapping file he's tested with. I don't think we should be modifying a file we've long used in the current form, especially when this doesn't reflect some operational change. This way there won't be any failures in the regression tests either.

How you'd choose the option for missing variables really depends. We never implemented any behavior to not write an array to file if "skip" is chosen and the variable is unavailable in the grib2 file. I believe the philosophy was always that, given a certain weather model SDF, we always wanted the atm init files to have certain arrays in them regardless of what data is available in the grib2 files used to create them. If I remember correctly, we use "skip" for those variables that might be missing data at certain levels, like RH in some GFS grib2 files, so that a few missing levels wouldn't cause a failure. If you plan on a variable possible being completely missing from a grib2 file but don't want that to cause a failure, I'd suggest using set_to_fill.

So, in that sort of paradigm, one might expect any weather model run utilizing HRRR Smoke to also require its own unique SDF, so it would make sense to have a separate variable mapping file. In that mapping file you could indeed set the missing behavior to be "stop" to make sure a user is initializing that weather model SDF with all the desired smoke tracers.

@jordanschnell
Copy link
Contributor Author

@LarissaReames-NOAA @GeorgeGayno-NOAA I have created a separate varmap file that contains smoke and removed smoke from the original file. To use, the varmap file in make_ics/make_lbcs scripts needs to be set to the new file. Please let me know if you would like further modifications.

@GeorgeGayno-NOAA
Copy link
Collaborator

@jordanschnell - be sure to merge updates from develop to your branch.

@jordanschnell
Copy link
Contributor Author

@jordanschnell - be sure to merge updates from develop to your branch.

@GeorgeGayno-NOAA, is that just:

git checkout rap_icbc
git merge develop

Then do I need to git push?

@GeorgeGayno-NOAA
Copy link
Collaborator

@jordanschnell - be sure to merge updates from develop to your branch.

@GeorgeGayno-NOAA, is that just:

git checkout rap_icbc git merge develop

Then do I need to git push?

That will work if 'develop' in your fork is up-to-date with the authoritative fork. See our wiki page for how I do it (last two bullets).

https://github.com/ufs-community/UFS_UTILS/wiki/3.-Checking-out-code-and-making-changes

I like to do things from the command line. But, I would guess you can also do these updates from the web browser.

@GeorgeGayno-NOAA
Copy link
Collaborator

Because you are adding a varmap file, this section of the documentation may need some updates:
https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html#variable-mapping-varmap-table

I can help with that.

@jordanschnell
Copy link
Contributor Author

@GeorgeGayno-NOAA I have access to Hera, but not the others. so, these are my remaining tasks?

  1. Update driver.hera script and test there?
  2. Add comments to regtest
  3. Add documentation

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA I have access to Hera, but not the others. so, these are my remaining tasks?

  1. Update driver.hera script and test there?
  2. Add comments to regtest
  3. Add documentation

And do as many of the tests described above in "TESTS CONDUCTED". I can do the tests on the Herc, Orion and WCOSS2.

@jordanschnell
Copy link
Contributor Author

Regtest on Hera passed: /scratch1/BMC/wrf-chem/Jordan/RRFS-SD/src/PRs/rap_icbc/rrfs-workflow/sorc/UFS_UTILS/reg_tests/chgres_cube/consistency.log14

@GeorgeGayno-NOAA
Copy link
Collaborator

@jordanschnell - I updated and tested the regression test driver scripts on Hercules, Orion and WCOSS2. I placed them on Hera here: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.schnell/reg_tests/chgres_cube

They have ".updated" in their filename.

@jordanschnell
Copy link
Contributor Author

@jordanschnell - I updated and tested the regression test driver scripts on Hercules, Orion and WCOSS2. I placed them on Hera here: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.schnell/reg_tests/chgres_cube

They have ".updated" in their filename.

Great thank you! So commit these?

@GeorgeGayno-NOAA
Copy link
Collaborator

@jordanschnell - I updated and tested the regression test driver scripts on Hercules, Orion and WCOSS2. I placed them on Hera here: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.schnell/reg_tests/chgres_cube
They have ".updated" in their filename.

Great thank you! So commit these?

Yes.

@GeorgeGayno-NOAA
Copy link
Collaborator

Because you are adding a varmap file, this section of the documentation may need some updates: https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html#variable-mapping-varmap-table

I can help with that.

@jordanschnell - Once you update the documentation, I will approve and merge this PR.

@jordanschnell
Copy link
Contributor Author

jordanschnell commented Nov 4, 2024

Because you are adding a varmap file, this section of the documentation may need some updates: https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html#variable-mapping-varmap-table
I can help with that.

@jordanschnell - Once you update the documentation, I will approve and merge this PR.

Great - how exactly do I update this?

As for specifically what to add, are you thinking I add a note under the Column 1, something to the effect of:

*** If smoke from RAP/HRRR is desired, one must select the GSDphys_smoke_var_map.txt varmap file which includes

massden - smoke mass mixing ratio


@GeorgeGayno-NOAA
Copy link
Collaborator

Because you are adding a varmap file, this section of the documentation may need some updates: https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html#variable-mapping-varmap-table
I can help with that.

@jordanschnell - Once you update the documentation, I will approve and merge this PR.

Great - how exactly do I update this?

As for specifically what to add, are you thinking I add a note under the Column 1, something to the effect of:

*** If smoke from RAP/HRRR is desired, one must select the GSDphys_smoke_var_map.txt varmap file which includes

massden - smoke mass mixing ratio

I set up readthedocs for your branch here:
https://schnel.readthedocs.io/en/rap_icbc/ufs_utils.html#variable-mapping-varmap-table

You will edit the ./docs/source/chgres_cube.rst file. When you push your branch to your fork, the new documentation should automatically update.

Yes, I would add the definition of massden to the list. Try that first and see if everything works. Then, we can add additional instructions for users.

@jordanschnell
Copy link
Contributor Author

jordanschnell commented Nov 4, 2024

@GeorgeGayno-NOAA - Commit is pushed, but I don't see it in the docs branch you created

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA - Commit is pushed, but I don't see it in the docs branch you created

Looks like I have to run it manually. Take a look now. The formatting does not look right.

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA - Commit is pushed, but I don't see it in the docs branch you created

Looks like I have to run it manually. Take a look now. The formatting does not look right.

I would just add the definition of massden.

@jordanschnell
Copy link
Contributor Author

@GeorgeGayno-NOAA - Commit is pushed, but I don't see it in the docs branch you created

Looks like I have to run it manually. Take a look now. The formatting does not look right.

I would just add the definition of massden.

ok hopefully that worked

Screenshot 2024-11-04 at 3 05 14 PM

@GeorgeGayno-NOAA
Copy link
Collaborator

The format looks good.

One more favor, can you remove the line: "(Note, only the GFS physics suite is currently supported.)" and add "Each file contains the following columns:"

@jordanschnell
Copy link
Contributor Author

The format looks good.

One more favor, can you remove the line: "(Note, only the GFS physics suite is currently supported.)" and add "Each file contains the following columns:"

Should be good to go!

@GeorgeGayno-NOAA
Copy link
Collaborator

PR #993 will be merged before this one. When that happens, you will need to merge those updates to your branch. PR #993 updates another program. So, there won't be any conflicts.

@GeorgeGayno-NOAA
Copy link
Collaborator

@jordanschnell - please merge the latest changes from 'develop' to your branch. I will then do a few final checks, then merge your changes.

@jordanschnell
Copy link
Contributor Author

@jordanschnell - please merge the latest changes from 'develop' to your branch. I will then do a few final checks, then merge your changes.

Should be good to go.

@GeorgeGayno-NOAA GeorgeGayno-NOAA merged commit a7f882f into ufs-community:develop Nov 12, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add smoke density from RAP files as an optional output from chgres
5 participants