Skip to content

UFS Global Weather Workflow

Jump to bottom
Rocky Dunlap edited this page Oct 4, 2019 · 33 revisions

This page describes a demonstration of the Unified Forecast System (UFS) Global Weather application with the Common Infrastructure for Modeling the Earth (CIME) workflow. CIME is being considered as a candidate for the upcoming UFS public release and the prototype here provides a working example of a three-step workflow including a pre-processing, forecast, and post-processing step.

This prototype workflow leverages recent capabilities added to CIME that support creating more sophisticated sequences of tasks, such as model runs that include separate pre- and post-processing jobs. These jobs can be submitted as separate jobs to a batch queuing system, including job dependencies, or a more robust workflow management system, such as Cylc.

In this demonstration workflow, the user will take several steps to download the FV3GFS code, create an experiment "case", build the code, and then submit the 3-step workflow. The pre-processor used is CHGRES and it will read in a fixed set of inputs and produce input files for FV3GFS in C96 resolution. Then a short global forecast is run using these inputs. After the forecast, the post-processor, UPP, is executed to produce grib2 output. All three workflow steps are automatically submitted to the batch system when the user runs a single command: case.submit.

Limitations and Known Bugs

Proof of concept implementation

  • This is a proof of concept implementation of the global workflow and has not undergone significant testing, and does not yet meet robustness requirements required for any software release. There are a number of known issues that were intentionally not addressed in order to rapidly deliver an end-to-end workflow suitable for a high-level evaluation.

Pre-processor, post-processor, and dependent libraries must be on the machine

  • This workflow assumes that the pre-processor, post-processor and their dependent libraries are already installed on the system. They are not downloaded or built with the model code. Significant effort was required to install all of the dependencies. Moreover, since building these would be time consuming, it is not recommended that all dependencies should be built using the workflow.

Only one model resolution and initial condition supported

  • When CIME executes the CHGRES pre-processor, a hard-coded, fixed set of inputs are used and a single model resolution is supported. Significant effort would be needed to provide flexible configuration options, such as switching to a new set of initial conditions inputs or changing the resolution. It is also unclear whether CIME should provide interfaces to the user to control these aspects, or if the user is expected to understand how CHGRES works, and manually modify the namelist. These same questions can be asked of the post-processor as well.

Several namelist files and inputs are hard coded

  • The namelist and some input files for CHGRES and UPP (gfs.post) are copied in from static directories and do not take into account grid resolutions, forecast period, etc. CIME needs to be extended to better support these namelist files.
    • CHGRES: The namelist file such as config.C96.nml is not handled by CIME yet.
    • FV3GFS: Even though CIME could handle diag_table, input.nml and model_configure files, the files are overwritten (defined as UGCSINPUTPATH in env_mach_specific.xml file) to be consistent with the pre- and post-processing tools. There is an issue that the changes in the diag_table could lead to error in post-processing step.
    • gfs.post: The namelist files such as postxconfig-NT.txt, params_grib2_tbl_new and itaga.* for post-processing is not handled by CIME yet.

Limited communication to user about workflow status

  • When the user runs the workflow, three jobs are submitted to the batch queuing system: pre-process, forecast, post-process. However, there is currently limited information provided to the user about the status of each job. A consistent mechanism is needed to communicate back to the user information about each job in the workflow.

Custom Python environment might cause error when running workflow

  • Currently, custom Python installation that has no required modules (os, sys etc.) and Conda environment that loads the Cheyenne python/3 module lead an error in CIME side and it prevents to run the workflow.

Supported Platforms

  • Cheyenne/NCAR

Prerequisites

  • The user must have access to NOAA's VLab with SSH keys set up. This is a temporary requirement and will go away when FV3GFS is on GitHub.

Download, Build and Run

To download, run and build:

$ git clone https://github.com/ESCOMP/UFSCOMP.git
$ cd UFSCOMP

# To checkout the global weather workflow application:
$ git checkout app_fv3gfs

# Set GIT_AUTHOR_NAME environment variable as Vlab user name
$ export GIT_AUTHOR_NAME=ufuk.turuncoglu

# Check out all model components and CIME
$ ./manage_externals/checkout_externals

# Go to CIME scripts directory
$ cd cime/scripts

# If it is different from the default one, set the PROJECT environment variable and replace PROJECT ID with an appropriate project number 
# Bourne shell (sh, ksh, bash, sh)
$ export PROJECT=[PROJECT ID]
# C shell (csh, tcsh)
$ setenv PROJECT [PROJECT ID]

# Create UFS Global Weather Workflow case using the name "fv3gfs_global_app" (the user can choose any name)
$ ./create_newcase --compset UFS_Weather --res C96 --case fv3gfs_global_app --workflow-case fv3gfs_global --driver nuopc --run-unsupported

# Setup case
$ cd fv3gfs_global_app
$ ./case.setup

# Build it (standalone FV3GFS)
$ ./case.build

# Change configuration options such as start time, wall clock limit for schedular, simulation time
$ ./xmlchange DOUT_S=FALSE
$ ./xmlchange STOP_OPTION=nhours
$ ./xmlchange STOP_N=36
$ ./xmlchange RUN_REFDATE=2016-10-03
$ ./xmlchange RUN_STARTDATE=2016-10-03
$ ./xmlchange JOB_WALLCLOCK_TIME=00:30:00
$ ./xmlchange USER_REQUESTED_WALLTIME=00:30:00

# Add following line to user_nl_cpl file
component_list = "ATM"

# Modify env_mach_specific.xml and add or change following items with appropriate ones
# On Cheyenne:
# <env name="UGCSINPUTPATH">/glade/scratch/turuncu/fv3gfs/fv3_gfdlmprad</env>
# <env name="CHGRES_NML">/glade/scratch/turuncu/fv3gfs/chgres.test/config.C96.nml</env>
# <env name="CHGRES_BIN">/glade/work/turuncu/UFS/UFS_UTILS/exec/chgres_cube.exe</env>
# <env name="NCEP_POST_BIN">/glade/work/turuncu/UFS/GLB_WRKF/sorc/gfs_post.fd/exec/ncep_post</env>
# <env name="LD_LIBRARY_PATH">/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/apps/ch/os/lib64:$ENV{LD_LIBRARY_PATH}</env>

# Note: You might need to copy following files to input directory (UGCSINPUTPATH)
# For chgres:
# - C96_mosaic.nc file into /INPUT
# For gfs.post:
# - postxconfig-NT.txt
# - params_grib2_tbl_new
# - itaga.012, itaga.024, itaga.036
# - nemsfile

# To submit simulation,
$ ./case.submit

FAQ

How does the CIME Case Control System (CCS) relate to existing build systems?

A complete modeling system such as the UFS has a lot of dependent software. This includes at least the dependent libraries, tools used for pre-processing and post-processing, and the model code itself. Most of the dependent libraries change infrequently and do not need to be re-built regularly and are typically installed in a single location available to all users. The same is typically true of tools such as a pre-processor, which can be installed and maintained by a system administrator, and then re-used by all users on a platform. The CIME CCS assumes that these kinds of dependent software are already installed on a system and have been built using their own native build systems. CIME CCS will not be used to build these parts of the code.

For the model code itself, each component model may have its own build system using make, autoconf, cmake, etc. That build system is linked to the CIME CCS with a file buildlib which provides the inputs to the component's build system and invokes it. In this way the CIME CCS controls when each model component is built and links it into the final executable.

Do I always need to rebuild the code as part of the workflow, even if I am cycling the same executable?

The CIME CCS will only rebuild code when it has changed, assuming that the underlying model components' build systems support this (i.e., if they are using a tool like make, which is aware of dependencies between source files). Once an executable has been generated, it can be run in a cycled fashion without rebuilding.

How can I add a new task to existing workflow?

The workflow called fv3gfs_global is defined in config_workflow.xml, which is under the cime/config/cesm/machines directory. To add new task to the existing workflow, you need to follow these steps:

  • Edit config_workflow.xml and add the new task

    The current workflow XML looks like following:

    ...
<workflow_jobs case="fv3gfs_global">
  <!-- order matters, with no-batch jobs will be run in the order listed here -->
  <job name="case.chgres">
    <template>template.chgres.run</template>
    <prereq>$BUILD_COMPLETE</prereq>
    <runtime_parameters>
      <task_count>36</task_count>
      <tasks_per_node>36</tasks_per_node>
      <walltime>0:20:00</walltime>
    </runtime_parameters>
  </job>
  <job name="case.run">
    <template>template.case.run</template>
    <dependency>case.chgres.run</dependency>
    <prereq>$BUILD_COMPLETE and not $TEST</prereq>
  </job>
  <job name="case.test">
    <template>template.case.test</template>
    <prereq>$BUILD_COMPLETE and $TEST</prereq>
  </job>
  <job name="case.gfs_post">
    <template>template.gfs_post.run</template>
    <dependency>case.run</dependency>
    <prereq>$BUILD_COMPLETE</prereq>
    <runtime_parameters>
      <task_count>8</task_count>
      <tasks_per_node>8</tasks_per_node>
      <walltime>0:20:00</walltime>
    </runtime_parameters>
  </job>
  <job name="case.st_archive">
    <template>template.st_archive</template>
    <!-- If DOUT_S is true and case.run (or case.test) exits successfully then run st_archive-->
    <dependency>case.run or case.test</dependency>
    <prereq>$DOUT_S</prereq>
    <runtime_parameters>
      <task_count>1</task_count>
      <tasks_per_node>1</tasks_per_node>
      <walltime>0:20:00</walltime>
    </runtime_parameters>
  </job>
</workflow_jobs>
...

    It includes case.chgres, case.run, case.test, case.gfs_post and case.st_archive tasks. For each task, the template entry specifies a template script that executes the specific task and dependency and prereq entries are used to define dependencies and prerequisites among the individual tasks.

    The following part shows how to add an example task (downloading input data for CHGRES) to the existing workflow. Note that some HPC platforms do not support connecting to the Internet from compute nodes. To that end, the new task defined here will not currently work on Cheyenne. An update to CIME is required to force this particular task to run on the head node, which can connect to the Internet to retrieve the data. Nonetheless, the example is still relevant for understanding how to add a workflow task.

    Add following task just before case.chgres:

      <job name="case.getdata">
    <template>template.getdata.run</template>
    <prereq>$BUILD_COMPLETE</prereq>
    <runtime_parameters>
      <task_count>1</task_count>
      <tasks_per_node>1</tasks_per_node>
      <walltime>0:20:00</walltime>
    </runtime_parameters>
  </job>

    In this case, new case.getdata task will run on single core to download data.

    Note that the job name used by the scheduler will be the combination of suffix after last dot symbol (.) and the case name. The case.getdata task will be show as getdata.fv3gfs_global_app in the job submission script.

    It is also necessary to add a dependency to case.chgres for case.getdata as follows:

      <job name="case.chgres.run">
    <template>template.chgres.run</template>
    <dependency>case.getdata</dependency>
    <prereq>$BUILD_COMPLETE</prereq>
    <runtime_parameters>
      <task_count>36</task_count>
      <tasks_per_node>36</tasks_per_node>
      <walltime>0:20:00</walltime>
    </runtime_parameters>
  </job>

  • Create the appropriate template script for the task

    The template.getdata.run script that is defined in the previous step downloads the most recent data from the NOMADS data server using the wget command.

    #!/bin/bash
{{ batchdirectives }}

# Goto run directory
rundir=`./xmlquery --value RUNDIR`
echo "run dir is $rundir"
cd $rundir

# Get list of files from NOMAD server
rm -f *.html
wget -O index.html https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod

# Retrive most recent one
idir=`cat index.html | grep 'gdas\.' | tail -n 1 | awk -F\" '{print $2}'`
wget "https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/${idir}00/gdas.t00z.sfcanl.nemsio"
wget "https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/${idir}00/gdas.t00z.atmanl.nemsio"

Now, the commands listed in the Download, Build and Run section can be used to run the workflow with the newly added task.

Note: In the current implementation, config.C96.nml file is not generated automatically by the CIME workflow and it is copied from the directory pointed by the $CHGRES_NML environment variable. To process new input with chgres, the config.C96.nml file needs to be edited and data_dir_input_grid namelist option must to point the run directory.

How can I see/check the steps in my workflow?

A good way to see what case.submit will do, is to first call preview_run, which will output the environment for your run along with the batch submit and mpirun commands. The following is example output for the fv3gfs_global workflow:

# Check out all model components and CIME
$ cd [CASE_DIRECTORY]
$ ./preview_run

CASE INFO:
  nodes: 5
  total tasks: 150
  tasks per node: 36
  thread count: 1

BATCH INFO:
  FOR JOB: case.getdata
    ENV:
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python purge 
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python load ncarenv/1.2 intel/19.0.2 esmf_libs mkl mpt/2.19 netcdf-mpi/4.6.1 pnetcdf/1.11.0 ncarcompilers/0.5.0
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=1024M
      Setting Environment TMPDIR=/glade/scratch/turuncu
      Setting Environment MPI_TYPE_DEPTH=16
      Setting Environment MPI_IB_CONGESTED=1
      Setting Environment ESMFMKFILE=/glade/work/turuncu/ESMF/8.0.0b48/lib/libO/Linux.intel.64.mpt.default/esmf.mk
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment UGCSINPUTPATH=/glade/scratch/turuncu/fv3gfs/fv3_gfdlmprad
      Setting Environment UGCSFIXEDFILEPATH=/glade/work/turuncu/FV3GFS/fix_am
      Setting Environment UGCSADDONPATH=/glade/work/turuncu/FV3GFS/addon
      Setting Environment CHGRES_NML=/glade/scratch/turuncu/fv3gfs/chgres.test/config.C96.nml
      Setting Environment CHGRES_BIN=/glade/work/turuncu/UFS/UFS_UTILS/exec/chgres_cube.exe
      Setting Environment NCEP_POST_BIN=/glade/work/turuncu/UFS/GLB_WRKF/sorc/gfs_post.fd/exec/ncep_post
      Setting Environment LD_LIBRARY_PATH=/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/opt/mpt_fmods/2.19/intel/19.0.2:/glade/u/apps/ch/opt/mpt/2.19/lib:/glade/u/apps/opt/intel/2019u2/mkl/lib/intel64:/glade/u/apps/opt/intel/2019u2/tbb/lib/intel64/gcc4.7:/glade/u/apps/opt/intel/2019u2/compilers_and_libraries/linux/lib/intel64:/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/u/home/turuncu/progs/esmf-7.1.0b30/install_dir/lib/libO/Linux.intel.64.intelmpi.default:/glade/u/home/turuncu/progs/python-2.7.16/lib:/ncar/opt/slurm/latest/lib:/glade/u/apps/ch/os/usr/lib64:/glade/u/apps/ch/os/usr/lib:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/os/lib
      Setting Environment TMPDIR=/glade/scratch/turuncu
      Setting Environment MPI_USE_ARRAY=false
      Setting Environment OMP_NUM_THREADS=1

    SUBMIT CMD:
      qsub -q share -l walltime=0:20:00 -A P93300606  .case.getdata

  FOR JOB: case.chgres
    ENV:
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python purge 
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python load ncarenv/1.2 intel/19.0.2 esmf_libs mkl mpt/2.19 netcdf-mpi/4.6.1 pnetcdf/1.11.0 ncarcompilers/0.5.0
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=1024M
      Setting Environment TMPDIR=/glade/scratch/turuncu
      Setting Environment MPI_TYPE_DEPTH=16
      Setting Environment MPI_IB_CONGESTED=1
      Unsetting Environment MPI_USE_ARRAY
      Setting Environment ESMFMKFILE=/glade/work/turuncu/ESMF/8.0.0b48/lib/libO/Linux.intel.64.mpt.default/esmf.mk
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment UGCSINPUTPATH=/glade/scratch/turuncu/fv3gfs/fv3_gfdlmprad
      Setting Environment UGCSFIXEDFILEPATH=/glade/work/turuncu/FV3GFS/fix_am
      Setting Environment UGCSADDONPATH=/glade/work/turuncu/FV3GFS/addon
      Setting Environment CHGRES_NML=/glade/scratch/turuncu/fv3gfs/chgres.test/config.C96.nml
      Setting Environment CHGRES_BIN=/glade/work/turuncu/UFS/UFS_UTILS/exec/chgres_cube.exe
      Setting Environment NCEP_POST_BIN=/glade/work/turuncu/UFS/GLB_WRKF/sorc/gfs_post.fd/exec/ncep_post
      Setting Environment LD_LIBRARY_PATH=/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/opt/mpt_fmods/2.19/intel/19.0.2:/glade/u/apps/ch/opt/mpt/2.19/lib:/glade/u/apps/opt/intel/2019u2/mkl/lib/intel64:/glade/u/apps/opt/intel/2019u2/tbb/lib/intel64/gcc4.7:/glade/u/apps/opt/intel/2019u2/compilers_and_libraries/linux/lib/intel64:/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/home/turuncu/progs/esmf-7.1.0b30/install_dir/lib/libO/Linux.intel.64.intelmpi.default:/glade/u/home/turuncu/progs/python-2.7.16/lib:/ncar/opt/slurm/latest/lib:/glade/u/apps/ch/os/usr/lib64:/glade/u/apps/ch/os/usr/lib:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/os/lib
      Setting Environment OMP_NUM_THREADS=1

    SUBMIT CMD:
      qsub -q regular -l walltime=0:20:00 -A P93300606  -W depend=afterok:0  .case.chgres

  FOR JOB: case.run
    ENV:
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python purge 
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python load ncarenv/1.2 intel/19.0.2 esmf_libs mkl mpt/2.19 netcdf-mpi/4.6.1 pnetcdf/1.11.0 ncarcompilers/0.5.0
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=1024M
      Setting Environment TMPDIR=/glade/scratch/turuncu
      Setting Environment MPI_TYPE_DEPTH=16
      Setting Environment MPI_IB_CONGESTED=1
      Setting Environment ESMFMKFILE=/glade/work/turuncu/ESMF/8.0.0b48/lib/libO/Linux.intel.64.mpt.default/esmf.mk
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment UGCSINPUTPATH=/glade/scratch/turuncu/fv3gfs/fv3_gfdlmprad
      Setting Environment UGCSFIXEDFILEPATH=/glade/work/turuncu/FV3GFS/fix_am
      Setting Environment UGCSADDONPATH=/glade/work/turuncu/FV3GFS/addon
      Setting Environment CHGRES_NML=/glade/scratch/turuncu/fv3gfs/chgres.test/config.C96.nml
      Setting Environment CHGRES_BIN=/glade/work/turuncu/UFS/UFS_UTILS/exec/chgres_cube.exe
      Setting Environment NCEP_POST_BIN=/glade/work/turuncu/UFS/GLB_WRKF/sorc/gfs_post.fd/exec/ncep_post
      Setting Environment LD_LIBRARY_PATH=/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/opt/mpt_fmods/2.19/intel/19.0.2:/glade/u/apps/ch/opt/mpt/2.19/lib:/glade/u/apps/opt/intel/2019u2/mkl/lib/intel64:/glade/u/apps/opt/intel/2019u2/tbb/lib/intel64/gcc4.7:/glade/u/apps/opt/intel/2019u2/compilers_and_libraries/linux/lib/intel64:/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/home/turuncu/progs/esmf-7.1.0b30/install_dir/lib/libO/Linux.intel.64.intelmpi.default:/glade/u/home/turuncu/progs/python-2.7.16/lib:/ncar/opt/slurm/latest/lib:/glade/u/apps/ch/os/usr/lib64:/glade/u/apps/ch/os/usr/lib:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/os/lib
      Setting Environment OMP_NUM_THREADS=1

    SUBMIT CMD:
      qsub -q regular -l walltime=12:00:00 -A P93300606  -W depend=afterok:1 -v ARGS_FOR_SCRIPT='--resubmit' .case.run

    MPIRUN (job=case.run):
      mpiexec_mpt -p "%g:"  -np 150  omplace -tm open64  /glade/scratch/turuncu/fv3gfs_global_app/bld/cesm.exe   >> cesm.log.$LID 2>&1 

  FOR JOB: case.gfs_post
    ENV:
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python purge 
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python load ncarenv/1.2 intel/19.0.2 esmf_libs mkl mpt/2.19 netcdf-mpi/4.6.1 pnetcdf/1.11.0 ncarcompilers/0.5.0
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=1024M
      Setting Environment TMPDIR=/glade/scratch/turuncu
      Setting Environment MPI_TYPE_DEPTH=16
      Setting Environment MPI_IB_CONGESTED=1
      Setting Environment ESMFMKFILE=/glade/work/turuncu/ESMF/8.0.0b48/lib/libO/Linux.intel.64.mpt.default/esmf.mk
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment UGCSINPUTPATH=/glade/scratch/turuncu/fv3gfs/fv3_gfdlmprad
      Setting Environment UGCSFIXEDFILEPATH=/glade/work/turuncu/FV3GFS/fix_am
      Setting Environment UGCSADDONPATH=/glade/work/turuncu/FV3GFS/addon
      Setting Environment CHGRES_NML=/glade/scratch/turuncu/fv3gfs/chgres.test/config.C96.nml
      Setting Environment CHGRES_BIN=/glade/work/turuncu/UFS/UFS_UTILS/exec/chgres_cube.exe
      Setting Environment NCEP_POST_BIN=/glade/work/turuncu/UFS/GLB_WRKF/sorc/gfs_post.fd/exec/ncep_post
      Setting Environment LD_LIBRARY_PATH=/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/opt/mpt_fmods/2.19/intel/19.0.2:/glade/u/apps/ch/opt/mpt/2.19/lib:/glade/u/apps/opt/intel/2019u2/mkl/lib/intel64:/glade/u/apps/opt/intel/2019u2/tbb/lib/intel64/gcc4.7:/glade/u/apps/opt/intel/2019u2/compilers_and_libraries/linux/lib/intel64:/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/home/turuncu/progs/esmf-7.1.0b30/install_dir/lib/libO/Linux.intel.64.intelmpi.default:/glade/u/home/turuncu/progs/python-2.7.16/lib:/ncar/opt/slurm/latest/lib:/glade/u/apps/ch/os/usr/lib64:/glade/u/apps/ch/os/usr/lib:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/os/lib
      Setting Environment TMPDIR=/glade/scratch/turuncu
      Setting Environment MPI_USE_ARRAY=false
      Setting Environment OMP_NUM_THREADS=1

    SUBMIT CMD:
      qsub -q share -l walltime=0:20:00 -A P93300606  -W depend=afterok:2  .case.gfs_post

  FOR JOB: case.st_archive
    ENV:
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python purge 
      module command is /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/libexec/lmod python load ncarenv/1.2 intel/19.0.2 esmf_libs mkl mpt/2.19 netcdf-mpi/4.6.1 pnetcdf/1.11.0 ncarcompilers/0.5.0
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=1024M
      Setting Environment TMPDIR=/glade/scratch/turuncu
      Setting Environment MPI_TYPE_DEPTH=16
      Setting Environment MPI_IB_CONGESTED=1
      Unsetting Environment MPI_USE_ARRAY
      Setting Environment ESMFMKFILE=/glade/work/turuncu/ESMF/8.0.0b48/lib/libO/Linux.intel.64.mpt.default/esmf.mk
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment UGCSINPUTPATH=/glade/scratch/turuncu/fv3gfs/fv3_gfdlmprad
      Setting Environment UGCSFIXEDFILEPATH=/glade/work/turuncu/FV3GFS/fix_am
      Setting Environment UGCSADDONPATH=/glade/work/turuncu/FV3GFS/addon
      Setting Environment CHGRES_NML=/glade/scratch/turuncu/fv3gfs/chgres.test/config.C96.nml
      Setting Environment CHGRES_BIN=/glade/work/turuncu/UFS/UFS_UTILS/exec/chgres_cube.exe
      Setting Environment NCEP_POST_BIN=/glade/work/turuncu/UFS/GLB_WRKF/sorc/gfs_post.fd/exec/ncep_post
      Setting Environment LD_LIBRARY_PATH=/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/opt/mpt_fmods/2.19/intel/19.0.2:/glade/u/apps/ch/opt/mpt/2.19/lib:/glade/u/apps/opt/intel/2019u2/mkl/lib/intel64:/glade/u/apps/opt/intel/2019u2/tbb/lib/intel64/gcc4.7:/glade/u/apps/opt/intel/2019u2/compilers_and_libraries/linux/lib/intel64:/glade/u/home/dunlap/YAML-INSTALL/lib:/glade/work/turuncu/UFS/REQ_LIBS/jasper-1.900.1/lib:/usr/lib64:/glade/u/home/turuncu/progs/esmf-7.1.0b30/install_dir/lib/libO/Linux.intel.64.intelmpi.default:/glade/u/home/turuncu/progs/python-2.7.16/lib:/ncar/opt/slurm/latest/lib:/glade/u/apps/ch/os/usr/lib64:/glade/u/apps/ch/os/usr/lib:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/os/lib
      Setting Environment TMPDIR=/glade/scratch/turuncu
      Setting Environment MPI_USE_ARRAY=false
      Setting Environment OMP_NUM_THREADS=1

    SUBMIT CMD:
      qsub -q share -l walltime=0:20:00 -A P93300606  -W depend=afterok:2 -v ARGS_FOR_SCRIPT='--resubmit' case.st_archive

How can I run an individual task in the exiting workflow?

The current implementation of the CIME workflow system allows you to define the first job submitted in a workflow by supplying the --job parameter to the case.submit command. CIME needs to be extended to include an option to run a single task from the workflow. The issue under CIME repository is opened for this new feature and can be found here.

An example use of the --job parameter:

$ ./case.submit --job case.chgres

In this case, the workflow will start from the case.chgres task and then complete all of the following tasks.

How can I define an entirely new workflow?

Defining a new workflow mainly includes similar steps to that of adding a new task to existing workflow. In this case, the whole workflow needs to be defined in the workflow XML file, considering all inter-dependencies of individual tasks. Each workflow must have a unique name.

How can I change wall clock time for specific task in the workflow?

This can be done by using xmlchange command as following

./xmlchange JOB_WALLCLOCK_TIME=00:10:00 --subgroup case.chgres

How can I change project account that will be used to submit jobs?

There are multiple ways to change project account that is used to submit job:

  • Set PROJECT environment variable before creating case

  • Use xmlchange command to change project account. For example, following command can be used to change project account for chgres task (please replace PROJECT ID with an appropriate project number).

    ./xmlchange CHARGE_ACCOUNT=[PROJECT ID] --subgroup case.chgres
Clone this wiki locally