Skip to content

Commit

Permalink
Reduce file size via 2-stream model initialization and DA cycling (#5)
Browse files Browse the repository at this point in the history 
This PR consists of 4 changes:
**1. Two-stream model initialization:**

Currently, we use the restart file for MPAS-JEDI cycling.  While we move to higher-resolution cycling experiments, we face the challenge of both memory usage and disk storage. This PR addresses the disk storage issue.

The idea is: instead of using restart files, we move to use the file (init.nc type background/analysis file) that keeps only the necessary fields.  Also split out static fields into a separate file.  These v7-based changes are based off of a v6-based branch (skamaroc@096b5d3).

**2. Allow comparable two-stream 'mpasout' cycling result with that of one-stream 'restart' cycling:**

This PR also fixes the large T2m/Q2m and low-level T/Q error growth issue that shows up in 120-km cycling experiments with this 2-stream workflow, which is caused by the different initialization of land use fields when config_do_restart is true and false. After this change in mpas_atmphys_landuse.F, using this two-stream code gives comparable cycling results to the original restart cycling.

**3. Add 4 more variables (pressure_base, pressure_p, u/v at cell center) in core_init_atmosphere/Registry.xml to ease init.nc file to be used for the model-space verification purpose.**

**4. Commented out one line in mpas_init_atm_cases.F to fix the over-specification issue of sea ice.**


Files changed:
M	src/core_atmosphere/Registry.xml
M	src/core_atmosphere/mpas_atm_core.F
M	src/core_atmosphere/mpas_atm_core_interface.F
M	src/core_atmosphere/physics/mpas_atmphys_landuse.F
M	src/core_init_atmosphere/Registry.xml
M	src/core_init_atmosphere/mpas_init_atm_cases.F

**Details for two-stream model initialization and DA cycling:**

- static stream: Includes the mesh, some of sfc_input variables(landmask, shdmin, albedo12m, etc) and parameters for gravity wave drag over orography.

- da_state stream: Fields are specified in the MPAS-Atmosphere Registry. Includes fields needed for DA purposes (either analysis variables or fixed input needed for CRTM or other obs operators)

- For cold start FC, both the static stream file and the input stream file should be set to the “init.nc” file produced by the init_atmosphere core;

- For cycling run (FC), the input stream file should be the new da_state stream file that was previously written by the model and modified/updated in the DA cycle. (Both static and da_state stream need to be specified in the streams.atmosphere file)

- For cycling run (DA): use static.nc to read in mesh fields (specified in the streams.atmosphere file) and set:           
    config_do_restart = false

- After implementing the capability, the size of the 120-km init.nc type background/analysis file is about 430M (double precision); ~50 variables (the size of original restart file is about 2GB). Memory usage also decreased when we separated the static stream: 
<img width="1256" alt="memory_usage" src="https://user-images.githubusercontent.com/23242703/90163085-dba06480-dd52-11ea-9d90-02ea7ffa5863.png">

- If users would like to use restart files in cycling DA, the only difference from before is in prepare “1stCycle_background” step, please see example: (pay attention to streams.atmosphere, namelist.atmosphere and run_fc2018041418.csh)
script: /glade/work/jban/pandac/JB01_restart_1stCycle_background/run_fc2018041418.csh
results:/glade/scratch/jban/pandac/JB01_restart_1stCycle_background

- The modification has no impact on restart mode run. The following figure shows 6-h forecast from restart cycling run (current code: old; this PR :new)  verified against GFSANA and there is no difference between them.
![JB01_restart-expmgfs_day0p25_NXTro_surface_pressure_RMS](https://user-images.githubusercontent.com/23242703/98064199-f690eb00-1e0e-11eb-8009-14070edeae57.png)

- Tested on Cheyenne. All ctests passed.
  • Loading branch information
@junmeiban @climbfuji
junmeiban authored and climbfuji committed Aug 25, 2022
1 parent aa8c606 commit bb2889b
Show file tree
Hide file tree
Showing 6 changed files with 143 additions and 27 deletions.
152 changes: 130 additions & 22 deletions src/core_atmosphere/Registry.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -395,9 +395,10 @@
<!-- **************************************************************************************** -->

<streams>
<stream name="input"
type="input"
filename_template="x1.40962.init.nc"

<stream name="static"
type="input"
filename_template="x1.40962.static.nc"
input_interval="initial_only"
immutable="true">

Expand Down Expand Up @@ -442,6 +443,10 @@
<var name="fEdge"/>
<var name="fVertex"/>
<var name="meshDensity"/>
<var name="meshScalingDel2"/>
<var name="meshScalingDel4"/>
<var name="meshScalingRegionalCell"/>
<var name="meshScalingRegionalEdge"/>
<var name="bdyMaskCell"/>
<var name="bdyMaskEdge"/>
<var name="bdyMaskVertex"/>
Expand All @@ -467,29 +472,73 @@
<var name="defc_a"/>
<var name="defc_b"/>
<var name="coeffs_reconstruct"/>
<var name="east"/>
<var name="north"/>
<var name="var2d"/>
<var name="con"/>
<var name="oa1"/>
<var name="oa2"/>
<var name="oa3"/>
<var name="oa4"/>
<var name="ol1"/>
<var name="ol2"/>
<var name="ol3"/>
<var name="ol4"/>
<var name="isltyp"/>
<var name="ivgtyp"/>
<var name="mminlu"/>
<var name="landmask"/>
<var name="shdmin"/>
<var name="shdmax"/>
<var name="snoalb"/>
<var name="albedo12m"/>
<var name="greenfrac"/>
<var name="pin"/>
<var name="ozmixm"/>
</stream>

<stream name="input"
type="input"
filename_template="x1.40962.init.nc"
input_interval="initial_only"
immutable="true">

<var_array name="scalars"/>
<var name="initial_time"/>
<var name="xtime"/>
<var name="cldfrac"/>
<var name="re_cloud" packages="mp_thompson_in;mp_wsm6_in"/>
<var name="re_ice" packages="mp_thompson_in;mp_wsm6_in"/>
<var name="re_snow" packages="mp_thompson_in;mp_wsm6_in"/>
<var name="u"/>
<var name="w"/>
<var name="rho"/>
<var name="pressure_p"/>
<var name="pressure_base"/>
<var name="theta"/>
<var name="relhum"/>
<var name="rho_base"/>
<var name="theta_base"/>
<var name="uReconstructZonal"/>
<var name="uReconstructMeridional"/>
<var name="surface_pressure"/>
<var name="isltyp"/>
<var name="ivgtyp"/>
<var name="mminlu"/>
<var name="refl10cm_max"/>
<var name="rainc"/>
<var name="rainnc"/>
<var name="lai"/>
<var name="isice_lu"/>
<var name="iswater_lu"/>
<var name="landmask"/>
<var name="shdmin"/>
<var name="shdmax"/>
<var name="snoalb"/>
<var name="albedo12m"/>
<var name="greenfrac"/>
<var name="sfc_albbck"/>
<var name="sfc_albedo"/>
<var name="sfc_albedo_seaice"/>
<var name="sfc_emibck"/>
<var name="mavail"/>
<var name="sfc_emiss"/>
<var name="thc"/>
<var name="ust"/>
<var name="xicem"/>
<var name="z0"/>
<var name="znt"/>
<var name="skintemp"/>
<var name="snow"/>
<var name="snowc"/>
Expand All @@ -506,19 +555,11 @@
<var name="t2m"/>
<var name="precipw"/>
<var name="dzs"/>
<var name="zs"/>
<var name="ter"/>
<var name="sh2o"/>
<var name="smois"/>
<var name="tslb"/>
<var name="var2d"/>
<var name="con"/>
<var name="oa1"/>
<var name="oa2"/>
<var name="oa3"/>
<var name="oa4"/>
<var name="ol1"/>
<var name="ol2"/>
<var name="ol3"/>
<var name="ol4"/>
<var name="h_oml_initial"/>
</stream>

Expand Down Expand Up @@ -1117,6 +1158,73 @@
<var_array name="lbc_scalars" />

</stream>

<stream name="da_state"
type="input;output"
filename_template="mpasout.$Y-$M-$D_$h.$m.$s.nc"
input_interval="initial_only"
output_interval="0_06:00:00"
immutable="true">

<var_array name="scalars"/>
<var name="initial_time"/>
<var name="xtime"/>
<var name="cldfrac"/>
<var name="re_cloud" packages="mp_thompson_in;mp_wsm6_in"/>
<var name="re_ice" packages="mp_thompson_in;mp_wsm6_in"/>
<var name="re_snow" packages="mp_thompson_in;mp_wsm6_in"/>
<var name="u"/>
<var name="w"/>
<var name="rho"/>
<var name="pressure_p"/>
<var name="pressure_base"/>
<var name="theta"/>
<var name="relhum"/>
<var name="rho_base"/>
<var name="theta_base"/>
<var name="uReconstructZonal"/>
<var name="uReconstructMeridional"/>
<var name="surface_pressure"/>
<var name="refl10cm_max"/>
<var name="rainc"/>
<var name="rainnc"/>
<var name="lai"/>
<var name="isice_lu"/>
<var name="iswater_lu"/>
<var name="sfc_albbck"/>
<var name="sfc_albedo"/>
<var name="sfc_albedo_seaice"/>
<var name="sfc_emibck"/>
<var name="mavail"/>
<var name="sfc_emiss"/>
<var name="thc"/>
<var name="ust"/>
<var name="xicem"/>
<var name="z0"/>
<var name="znt"/>
<var name="skintemp"/>
<var name="snow"/>
<var name="snowc"/>
<var name="snowh"/>
<var name="sst"/>
<var name="tmn"/>
<var name="vegfra"/>
<var name="seaice"/>
<var name="xice"/>
<var name="xland"/>
<var name="u10"/>
<var name="v10"/>
<var name="q2"/>
<var name="t2m"/>
<var name="precipw"/>
<var name="dzs"/>
<var name="zs"/>
<var name="ter"/>
<var name="sh2o"/>
<var name="smois"/>
<var name="tslb"/>
<var name="h_oml_initial"/>
</stream>
</streams>


Expand Down
4 changes: 3 additions & 1 deletion src/core_atmosphere/mpas_atm_core.F
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,16 +92,18 @@ function atm_core_init(domain, startTimeStamp) result(ierr)
if (config_do_restart) then
call MPAS_stream_mgr_read(domain % streamManager, streamID='restart', ierr=ierr)
else
call MPAS_stream_mgr_read(domain % streamManager, streamID='static', whence=MPAS_STREAM_NEAREST, ierr=ierr)
call MPAS_stream_mgr_read(domain % streamManager, streamID='input', ierr=ierr)
end if

if (ierr /= MPAS_STREAM_MGR_NOERR) then
call mpas_log_write('********************************************************************************', messageType=MPAS_LOG_ERR)
call mpas_log_write('Error reading initial conditions', messageType=MPAS_LOG_ERR)
call mpas_log_write('********************************************************************************', messageType=MPAS_LOG_CRIT)
end if
call MPAS_stream_mgr_reset_alarms(domain % streamManager, streamID='static', direction=MPAS_STREAM_INPUT, ierr=ierr)
call MPAS_stream_mgr_reset_alarms(domain % streamManager, streamID='input', direction=MPAS_STREAM_INPUT, ierr=ierr)
call MPAS_stream_mgr_reset_alarms(domain % streamManager, streamID='restart', direction=MPAS_STREAM_INPUT, ierr=ierr)

!
! Read all other inputs
! For now we don't do this here to match results with previous code; to match requires
Expand Down
2 changes: 1 addition & 1 deletion src/core_atmosphere/mpas_atm_core_interface.F
View file
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ function atm_get_mesh_stream(configs, stream) result(ierr)
else if (config_do_restart) then
write(stream,'(a)') 'restart'
else
write(stream,'(a)') 'input'
write(stream,'(a)') 'static'
end if

end function atm_get_mesh_stream
Expand Down
6 changes: 4 additions & 2 deletions src/core_atmosphere/physics/mpas_atmphys_landuse.F
View file
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ subroutine landuse_init_forMPAS(dminfo,julday,mesh,configs,diag_physics,sfc_inpu
integer,intent(in):: julday

!local pointers:
logical,pointer:: config_do_restart, &
logical,pointer:: config_do_restart, config_do_DAcycling, &
config_frac_seaice, &
config_sfc_albedo

Expand Down Expand Up @@ -133,6 +133,7 @@ subroutine landuse_init_forMPAS(dminfo,julday,mesh,configs,diag_physics,sfc_inpu
!call mpas_log_write('--- enter subroutine landuse_init_forMPAS:')

call mpas_pool_get_config(configs,'config_do_restart' ,config_do_restart )
call mpas_pool_get_config(configs,'config_do_DAcycling' ,config_do_DAcycling )
call mpas_pool_get_config(configs,'config_frac_seaice',config_frac_seaice)
call mpas_pool_get_config(configs,'config_sfc_albedo' ,config_sfc_albedo )

Expand Down Expand Up @@ -252,8 +253,9 @@ subroutine landuse_init_forMPAS(dminfo,julday,mesh,configs,diag_physics,sfc_inpu
!call mpas_log_write('--- isice =$i',intArgs=(/isice/))
!call mpas_log_write('--- iswater =$i',intArgs=(/iswater/))
!call mpas_log_write('--- isurban =$i',intArgs=(/isurban/))
if(config_do_restart) then
if(config_do_restart .or. config_do_DAcycling) then
call mpas_log_write('--- config_do_restart =$l', logicArgs=(/config_do_restart/))
call mpas_log_write('--- config_do_DAcycling =$l', logicArgs=(/config_do_DAcycling/))
call mpas_log_write('--- skip the end of landuse_init_forMPAS')
return
endif
Expand Down
4 changes: 4 additions & 0 deletions src/core_init_atmosphere/Registry.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -555,6 +555,10 @@
<var name="relhum" packages="met_stage_out"/>
<var name="rho_base" packages="met_stage_out"/>
<var name="theta_base" packages="met_stage_out"/>
<var name="pressure_base" packages="met_stage_out"/>
<var name="pressure_p" packages="met_stage_out"/>
<var name="uReconstructZonal" packages="met_stage_out"/>
<var name="uReconstructMeridional" packages="met_stage_out"/>
<var name="surface_pressure" packages="met_stage_out"/>
<var name="precipw" packages="met_stage_out"/>
<var name="h_oml_initial" packages="met_stage_out"/>
Expand Down
2 changes: 1 addition & 1 deletion src/core_init_atmosphere/mpas_init_atm_cases.F
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4498,7 +4498,7 @@ subroutine init_atm_case_gfs(block, mesh, nCells, nEdges, nVertLevels, fg, state
if (allocated(maskslab)) deallocate(maskslab)
! Freeze really cold ocean
where (sst < 271.0 .and. landmask == 0) xice = 1.0
!where (sst < 271.0 .and. landmask == 0) xice = 1.0
! Limit XICE to values between 0 and 1. Although the input meteorological field is between 0.
! and 1., interpolation to the MPAS grid can yield values of XiCE less than 0. and greater
Expand Down

0 comments on commit bb2889b

Please sign in to comment.