WW3 ICs are not read for HR like experiements. #3109
Comments
JessicaMeixner-NOAA
added
bug
|
It appears that C48_S2SW CI test will demonstrate the issue. |
|
Okay - so running the C48_S2SW CI test can replicate this issue some log files here: Stage_IC does:
Fcst job goes into this section, it's a cold start and then just simply states: However, many times we have staged IC, so I don't quite understand this logic or when this changed happened. Still trying to go back and understand that. @aerorahul @WalterKolczynski-NOAA @KateFriedman-NOAA or others are you aware of why this is doing this? Some PRs of potential reference: For GEFS @sbanihash @NeilBarton-NOAA is there a CI test that has staged wave ICs that I can run to make sure I don't break that with these updates. |
@WalterKolczynski-NOAA - where are the ICs copied then? I checked the actual output of the wave model and there is no wave IC being used, so it's not just an erroneous error message -- this is a huge problem. In discussion with @NeilBarton-NOAA @sbanihash - we think it'd be a good idea to let the user chose if they want to error out if there is no wave IC or not and control that by a flag so that people can perhaps have an easier way of knowing about this. I think GEFS is okay b/c it's a warm start versus the tests I'm using which are cold starts which put you in different parts of the loop, but we'll continue to check. |
|
Okay, I see the problem now. Should be an easy fix. |
|
@WalterKolczynski-NOAA - I've been working on a fix for this, however it'll be great if you take this over. If you will be working on this, can you let me know what your tlimeline for a fix will be? |
I'm going to do it today. |
|
@WalterKolczynski-NOAA great thanks! Just to share where I was at, I have local changes here: /scratch1/NCEPDEV/climate/Jessica.Meixner/WaveICIssue/global-workflow which I tested here: /scratch1/NCEPDEV/climate/Jessica.Meixner/WaveICIssue/test02 and they did not work because this did not match the stage_ic location. I'm not sure if this means that the stage IC job should also get updated to be more in sync w/where a "warm" start IC would be expected or if I did not update things correctly on my end or made another mistake -but thought I'd share where I was at. |
Wave restart files were not being copied into the run directory for cold starts. Additionally, the previous restart directory used as the source for wave restarts (for non-RERUN) was always looking to the gdas RUN for gfs runs, which I do not believe is correct for waves since there is no DA. Resolves NOAA-EMC#3109
The stage job was incorrectly putting wave restarts into the gfs directory. The forecast job looks for them in the gdas directory, so this is updated. Additionally, the restarts were also not being copied from the staged directory to `$DATA`, so now they are. The process is identical to that of non-RERUN warm starts, so the code is re- factored a bit to avoid duplication. Resolves NOAA-EMC#3109
Adds a new vesion file for IC directories. Unlike other version files, this one uses an associative array instead of different variables. With the version file in place, the versions are updated on most of the directories to switch to the relocated wave restarts. Refs: NOAA-EMC#3109
Adds a new vesion file for IC directories. Unlike other version files, this one uses an associative array instead of different variables. With the version file in place, the versions are updated on most of the directories to switch to the relocated wave restarts. Refs: NOAA-EMC#3109
What is wrong?
The gfs_stage_ic job copies wave ICs:
^[[38;21m2024-11-17 13:45:38,610 - INFO - file_utils : Copied /lfs/h2/emc/couple/noscrub/jessica.meixner/WaveUglo15km/ICDIR/Opt3/gfs.20200912/18/model/wave/restart/20200913.000000.restart.ww3 to /lfs/h2/emc/couple/noscrub/jessica.meixner/WaveUglo15km/Opt3Sept/COMROOT/Opt3Sept/gfs.20200912/18//model/wave/restart^[[0mBut then the forecast job does not find ICs:
What should have happened?
ICs should be used in the HR prototype-like cases.
What machines are impacted?
All or N/A
What global-workflow hash are you using?
Techncially it's a branch, but this also appears to have happend in HR4 tag
Steps to reproduce
I'm running my branch here: https://github.com/JessicaMeixner-NOAA/global-workflow/tree/feature/uglo_15km
and staging my own IC directory.
So from: /lfs/h2/emc/couple/noscrub/jessica.meixner/WaveUglo15km/global-workflow/workflow
using the script coupled.sh to set up experiments.
Additional information
I checked HR4 output and there are no wave ICs as HS is 0 at the first output (haven't confirmed from log though).
Have pinged @sbanihash to be sure to check GEFS recent experiment results.
Have not yet checked low-res Ci tests, but there's likely an example use-case that could be used for debugging.
Do you have a proposed solution?
Trying to figure out logic https://github.com/NOAA-EMC/global-workflow/blob/develop/ush/forecast_postdet.sh#L330-L359 and in the stage_ic job to figure out where the disconnect is.
The text was updated successfully, but these errors were encountered: