Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for CMEC driver #2

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

acordonez
Copy link

@acordonez acordonez commented Jun 14, 2021

Summary:
The changes in this PR make CyMeP compatible with cmec-driver while preserving the existing functionality.

Additions:
settings.json is a settings file for cmec-driver.

coastal-storm-metrics.sh is a driver script for cmec-driver.

cymep.yml is an environment file that can be used to install the environment for running CyMeP in cmec-driver.

cymep.py can use CMEC environment variables to locate input and output folders. User settings can be loaded from the cmec.json configuration file. Descriptions are added for csv, netcdf, and figure outputs, which are written to a JSON called "output.json" as part of the cmec-driver output bundle.

Figure file type is changed to a user input in the NCL files. cmec-driver outputs png images for viewing in web browers, but graphics-cymep.sh still outputs pdfs. When calling one of the figure scripts, add out_type="<figure extension>" to the script call.

Usage:
Instructions are added to the README for registering and running the package with cmec-driver.

TODO:

  • Testing
  • Document cmec-driver specific changes for future developers

Note: Some of the plotting scripts (especially plot-taylor.ncl) show many changes in the diff because my editor automatically removed trailing whitespace.

@acordonez
Copy link
Author

SUMMARY OF CHANGES

README.md
Add instructions for running the package with cmec-driver.

coastal-storm-metrics.sh
Driver script that is called by cmec-driver. If the name of this script is changed, it must be updated in settings.py under the "driver" key.

settings.json
CMEC driver settings file. Contains information about the driver, input data, and runtime environment.

cymep.yml
This file can be used to create a CyMeP environment called _CMEC_cymep.

cymep/cymep.py
A bulleted list of changes:

  • Directory names are variables rather than hard-coded, but the defaults are the same as v1. These variables will be overwritten by the CMEC driver environment variables if present.
  • A "cmec-driver" flag is added. It becomes "true" if the cmec driver environment variable "CMEC_CONFIG_DIR" is set.
  • User settings are loaded from the cmec-driver configuration JSON if "CMEC_CONFIG_DIR" is set. These settings override the defaults in cymep.py.
  • Descriptions of the metrics and figures are written to output.json if cmec-driver=True.
  • The optional arguments "desc" and "cmec_driver" are passed to write_single_csv.
    -- "desc" is a 3-string list containing a unique short name, a long name, and a short description for each csv.
  • The optional arguments "wkdir" and "cmec_driver" are now passed to write_spatial_netcdf().

cymep/functions/getSettings.py
This function loads settings from a JSON configuration file (used to load them from the cmec-driver config file). These are the same variables from the "User settings" part of the cymep.py script. Called in cymep.py.

cymep/functions/write_cmec.py
Summary:

  • Adds descriptions for ncl-generated figures to output.json
  • Creates html page

Important functions:
define_figures():
If new figure are added to cymep, descriptions should be added to this function.

html_template():
Change the html template here. It is stored as a list of strings.

populate_html_figures(), populate_html_json():
Add the figures and metrics to the html template.

cymep/functions/write_spatial.py
Cmec-driver creates a file called "output.json" that describes all of the package output files. I've added functions to create and update this file throughout write_spatial.py

New and altered functions:
define_statistics():
If new metrics are added to the package, descriptions must be added to the dictionary in this function.

create_output_json():
Add or remove keys from output.json here.

write_spatial_netcdf():
I've added two optional keys to the function call: wkdir='.',cmec=False. If cmec=True, this function then writes a description to output.json and creates a JSON version of the metrics inside the netcdf.

write_nc_metrics_jsons():
If new metrics are added to (or removed from) the netcdf bundle, it may be necessary to add a JSON conversion here. The monthly, yearly, and taylor statistics are all outputed as JSON.

write_single_csv():
I've added two optional keys to the function call: wkdir='.',cmec=False. If cmec=True, this function then writes a description to output.json and creates a JSON version of the metrics inside the csv.

cymep/plotting/plot-*.ncl
A bullet list of changes:

  • Check the environment for CMEC_WK_DIR, otherwise set working directory to "."
  • Turn "out_type" into a runtime variable

cymep/graphics-cymep.sh
Add "output=pdf" to ncl script calls

@acordonez
Copy link
Author

@zarzycki These changes are working with the newest version of cmec-driver. It still depends on having post-processed trajectories (is not yet an end-to-end pipeline starting from model outputs). I've thrown together all of my changes in this PR, but let me know if you want things broken up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant