[Bug] 2-stream mpas_atmosphere warnings

[jjguerrette]

Current behavior (describe the bug)

So far only tested with a 120km domain.

When using the 2-stream (static+input) initial state in an mpas_atmosphere forecast within mpas-bundle, the following warnings are issued:

WARNING: Variable meshScalingDel2 not in input file.
WARNING: Variable meshScalingDel4 not in input file.
WARNING: Variable meshScalingRegionalCell not in input file.
WARNING: Variable meshScalingRegionalEdge not in input file.
WARNING: Variable east not in input file.
WARNING: Variable north not in input file.
WARNING: Variable pin not in input file.
WARNING: Variable ozmixm not in input file.

The static and and input streams are as follows in streams.atmosphere:

<immutable_stream name="static"
                  type="input"
                  filename_template="static.nc"
                  io_type="pnetcdf,cdf5"
                  input_interval="initial_only" />

<immutable_stream name="input"
                  type="input"
                  filename_template="mpasin.$Y-$M-$D_$h.$m.$s.nc"
                  io_type="pnetcdf,cdf5"
                  input_interval="initial_only" />

I am using an x1.40962.init.DATE.nc file linked to static.nc and a da_state (mpasout) file as mpasin.$Y-$M-$D_$h.$m.$s.nc.

Looking at src/core_atmosphere/Registry.xml, the static stream is supposed to include east, north, pin, and ozmixm. However, the output stream in src/core_init_atmosphere/Registry.xml, which is used to produce x1.40962.init.DATE.nc files, does not include any of those four variables. So either we should not use init.nc files as static stream inputs or one of the two Registry.xml variable lists needs to be modified.

The source of the other warning (meshScalingDel2, meshScalingDel4, meshScalingRegionalCell, and meshScalingRegionalEdge) is not clear. These are only listed in the restart stream of src/core_atmosphere/Registry.xml, but I have config_do_restart = false set in namelist.atmosphere. Also, those appear to be dependent variables set in src/core_atmosphere/mpas_atm_core::atm_compute_mesh_scaling, not variables that need to be read in.

To Reproduce

What computer are you running on?
Cheyenne

What compilers/modules are you using?
jedi/gnu-openmpi/9.1.0-v0.4

Steps to reproduce the behavior

1. Run mpas_atmosphere forecast in 2-stream workflow
2. Look at log.atmosphere.0000.out

Expected behavior

The warnings should not be issued, either because the fields are present in the static.nc file or because they are not needed and are removed from the requested variables.

Additional information (optional)

[jjguerrette]

east and north are calculated in init_dirs_forphys. So long as that subroutine is called when config_do_DACycling is true, then those two variables do not need to be in the static stream.

EDIT: the relevant subroutine tree is as follows
atm_mpas_init_block --> physics_init --> init_dirs_forphys

There are no logical tests that avoid the tree from being followed. Therefore east and north can be removed from the static stream.

[liujake]

@junmeiban Can you look into this unwanted warning issue? Ideally, it will also be cleaner if we can really use a 'static' file instead of an init file for the static stream to avoid confusion.

@syha Your originally implemented this two-stream initialization. What is the reason you used an 'init' file for the static stream? Do you remember what fields are missing in the normal static file to be used in this two-stream initialization?

[junmeiban]

Sure. I can look into it.

[liujake]

From my recent regional test using the 2-stream option, I noticed that mpas-jedi seems to write out the analysis file with many variables (controlled by stream_list.atmosphere_output?). This behavior is different from earlier version of code only writing our very few analysis variables (we used ncks or overwrote the background file via yaml setting). I remember @jjguerrette or @byoung-joo changed to have a more flexible/controllable I/O with a PR in mpas-jedi. Can you point which PR it is (I had hard time to find it) or point to the code for analysis I/O in mpas-jedi? I think this WARNING issue is more likely a mpas-jedi issue, not a MPAS-Model issue, understanding current analysis I/O in MPAS-JEDI will be helpful to solve it.

[byoung-joo]

From my recent regional test using the 2-stream option, I noticed that mpas-jedi seems to write out the analysis file with many variables (controlled by stream_list.atmosphere_output?).

Currently, all of the read and write functions in mpas-jedi are controlled by output stream, and the variables for 'read' and 'write' are listed in stream_list.atmosphere_output.

I remember @jjguerrette or @byoung-joo changed to have a more flexible/controllable I/O with a PR in mpas-jedi.

We have an issue for that, but it is not done yet: https://github.com/JCSDA-internal/mpas-jedi/issues/355 .
This might be a right time to do this...

Can you point which PR it is (I had hard time to find it) or point to the code for analysis I/O in mpas-jedi?

read : https://github.com/JCSDA-internal/mpas-jedi/blob/1a8887d9d43c04bf7ffa10a4a9d63bf77def9725/src/mpasjedi/mpas_field_utils_mod.F90#L320
write : https://github.com/JCSDA-internal/mpas-jedi/blob/1a8887d9d43c04bf7ffa10a4a9d63bf77def9725/src/mpasjedi/mpas_field_utils_mod.F90#L432

[liujake]

My impression is that variables to 'read' is not only controlled by the '*_output' stream, yaml setting for background state variables also matters. For example, I remember I added t2m/q2m in yaml for making rttov operator work, but seems to no need to add them in 'output' stream file. That is another confusion part to me (i.e., variables I/O vs. background/state variables in use).

[byoung-joo]

If t2m/q2m is not added to stream_list.atmosphere_output, then the values for t2m/q2m variables must be assigned at the mpas_init stage, when the mpas initial fields are read from init or restart or 2stream mpasout files. Not through the read function.

@byoung-joo It is not very clear to me from your statement: is it a problem or not if I do not add t2m/q2m in stream_list.atmosphere_output? Sounds like all variables will be read anyway by 'mpas_init' no matter variables are included or not in stream_list.atmosphere_output. Then I guess it is Ok to not have t2m/q2m in stream_list.atmosphere_output. Now I recall that there are two Input/Read steps: one via model's mpas_init (for every thing, we have no much control), one through mpas-jedi's 'read' function (for variables defined in stream_list.atmosphere_output). We have added code to deallocate unused variables loaded by mpas_init. This worked fine so far, but be always confusing when doing configuration.

Sorry. I accidently deleted some message by using edit mode instead of Quote reply mode.

[syha]

Jake, I checked my MPAS codes w/ the 2-stream option, and found that I had defined three different stream names for the purpose of separating out the streams: "static" for static.nc, "init" for init.nc, and "da_state" for analysis variables only. I don't know how you guys adopted my codes, but I defined and used static and init streams separately. Let me attach my src/core_atmosphere/Registry.xml here.
Registry.xml.txt

[liujake]

@syha We did the same as you, just have a slightly different set of variables for 'da_state' stream. You may check back, linking init.nc to static.nc is in your documentation on how to use this option, we just followed. I believe it will not work if we just use static.nc for static stream. My question is that what is the reason you did this way (i.e., linking init.nc to static.nc)? Any suggestions how to make this cleaner, i.e., really use static.nc file for static stream.

[liujake]

@syha #5 is the PR in this repo for 2-stream modification.

[jamiebresch]

When using the 2-stream (static+input) initial state in an mpas_atmosphere forecast within mpas-bundle, the following warnings are issued:

WARNING: Variable meshScalingDel2 not in input file.
WARNING: Variable meshScalingDel4 not in input file.
WARNING: Variable meshScalingRegionalCell not in input file.
WARNING: Variable meshScalingRegionalEdge not in input file.
WARNING: Variable east not in input file.
WARNING: Variable north not in input file.
WARNING: Variable pin not in input file.
WARNING: Variable ozmixm not in input file.

Info about those fields were specifically noted in the original message in skamaroc@096b5d3

[syha]

Jake, that document was written by Bill Skamarock. So was the cold-start part in the document.
I only cared about the restart mode, so this question should go to Bill who designed it that way for init.nc.

[liujake]

@syha I am confused. I thought you did this for saving disk space for your DART runs. So you never used this option for your DART runs? If you only care/use restart mode, you do not need to have this. So this option was originally designed/developed by Bill for cold-start mode?

[syha]

Just for clarification and as the record for everybody, by definition, static.nc contains horizontal fields only. In other words, no vertical info in there (e.g., config_vertical_grid = false). But for the model integration, we do need vertical mesh info such as zgrid, zz, etc. That's why, in the 2-stream version, you always need to get those variables from init.nc, which doesn't vary with time (e.g., static).

[junmeiban]

@syha thank you for clarification.

[junmeiban]

For the warning message,

• meshScalingDel{2,4}, meshScalingRegionalCell/Edge, east, and north fields  can be removed from static stream

• The key thing is how to deal with "pin and ozmixm" fields which mentioned in skamaroc/MPAS-Model@096b5d3: copy and paste here:


The pin and ozmixm fields have been moved to the "static" stream, but these are only available after the model starts up and reads in the OZONE_DAT.TBL and other files. Some cleverness is needed to get these fields into the file used for the "static" stream.


• according to code: if we use 2-stream, these two fields (pin and ozmixm) will be always calculated in core_atmosphere/physics/mpas_atmphys_init.F(
  (if(config_o3climatology .and. (.not. config_do_restart)  call init_o3climatology) 


• So I am wondering why skamaroc/MPAS-Model@096b5d3: still keep these two fields in static stream?

[jamiebresch]

* So I am wondering why [skamaroc/MPAS-Model@096b5d3](https://github.com/skamaroc/MPAS-Model/commit/096b5d3): still keep these two fields in static stream?

The implementation for jedi application (#5) is different from the original code https://github.com/skamaroc/MPAS-Model/tree/atmosphere/split_da_restart_files

[junmeiban]

@jamiebresch, thank you.

[jjguerrette]

@byoung-joo It is not very clear to me from your statement: is it a problem or not if I do not add t2m/q2m in stream_list.atmosphere_output? Sounds like all variables will be read anyway by 'mpas_init' no matter variables are included or not in stream_list.atmosphere_output. Then I guess it is Ok to not have t2m/q2m in stream_list.atmosphere_output.

@liujake, I just saw your "edit" message on @byoung-joo's post. I recommend that you add t2m/q2m to stream_list.atmosphere_output if you want the correct behavior. All real valued variables that are duplicated from the geometry to the state in mpas4da_mod::da_template_pool are set to zero after being copied. Only those variables in stream_list.atmosphere_output will be read from the background file in order to populate the State->mpas_fields%subFields pool. I would be very surprised if you have realistic values for t2m/q2m in the State->mpas_fields%subFields pool if you have not added those variables to the stream list. That pool is used to convert from model variables to geovars (GeoVaLs variables).

[liujake]

I see @jjguerrette . Will do in future runs. t2m/q2m should be added in streams.list file as part of standard variables.

[jjguerrette]

t2m/q2m should be added in streams.list file as part of standard variables

While that is true for RTTOV runs, they do not need to be in stream_list.atmosphere for non-RTTOV runs and would very slightly increase memory overhead. The best way to fix the issue is to define different stream_list.atmosphere files for different configurations. This is easily accomplished by having two stream_list.atmosphere.output files specified in two different versions of streams.atmosphere, then specifying two different streams_file values in the yaml for RTTOV and non-RTTOV simulations.

[jjguerrette]

Also see BJ's comment here: https://github.com/JCSDA-internal/mpas-jedi/pull/534#discussion_r600740647

In short, I was wrong about t2m and q2m. Their values are copied from the Geometry domain instance. However, it would be better to read the values from the background file, not from the Geometry. There is no guarantee that the same file is used to initialize the background and the Geometry domain instance. This is especially true in EDA, where the same netcdf file is used to fill in the field values of the Geometry domain pools for all members, but each member has its own background file.