Ensemble members do not recognize breakpoints

[benjamin-cash]

What is wrong?

Running with multiple ensemble members and breakpoints, but only mem000 appears to have recognized the breakpnt and stopped at the appropriate forecast hour.

What should have happened?

All members of the ensemble should have stopped at fhour 1472, which is when the breakpoint was set for. However, only mem000 did so - other members continued past the breakpoint and timed out.

What machines are impacted?

All or N/A

What global-workflow hash are you using?

077ad5f, from https://github.com/NeilBarton-NOAA/global-workflow/tree/SFS

Steps to reproduce

Generate ensemble workflow with breakpoints via https://github.com/NeilBarton-NOAA/global-workflow/tree/SFS
Run at least two members

Additional information

Log files from the successful mem000 and failed mem001 are too large to upload to github, so I have moved them to Hercules: /work/noaa/nems/cash/breakpoint_issue

Do you have a proposed solution?

No response

[benjamin-cash]

Comparing the two log files, in mem000 it looks like breakpnt hours is being used in forecast_predet.sh (log line 6358), but in mem001 it is the FHMAX_GFS value (log line 6769).

[WalterKolczynski-NOAA]

Please retry with current develop. This was fixed in #3009

[benjamin-cash]

Will do, thanks Walter. :)

[WalterKolczynski-NOAA]

@benjamin-cash Did this work with the more recent hash?

[benjamin-cash]

Hi @WalterKolczynski-NOAA - I have to look back at my notes to be sure, but I think I haven't been able to test it yet. I upgraded my whole experiment suite to a more recent version of the SFS branch and the SKEB settings caused all of my ensemble members to blow up almost immediately. I didn't think to check the log files to see if the breakpoints were being correctly identified while that was happening. Assuming the scrubber on Frontera didn't get them I'll take a look today and see.

[benjamin-cash]

@WalterKolczynski-NOAA - I can confirm now, from the logs at least, that the ensemble members are picking up the breakpoints. Thanks!