Merge pull request #170 from NOAA-CSL/develop

Add in new capabilities: Regulatory Options, Saving output, CLI tools, Filtering, etc.

.github/workflows/ci.yml

@@ -23,7 +23,7 @@ jobs:
       - uses: actions/checkout@v3
 
       - name: Set up Python (micromamba)
-        uses: mamba-org/provision-with-micromamba@v12
+        uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-file: docs/environment-docs.yml
           cache-env: true
@@ -67,7 +67,7 @@ jobs:
           cd docs/examples
           melodies-monet --version
           python -m melodies_monet --version
-          melodies-monet control_idealized.yaml
+          melodies-monet run control_idealized.yaml
           cd -
 
   docs:

docs/api.rst

@@ -1,13 +1,24 @@
 API
 ---
 
+.. module:: melodies_monet
+
+:mod:`melodies_monet` has these top-level submodules, which are imported lazily:
+
+.. currentmodule:: melodies_monet
+
 .. autosummary::
    :toctree: api/
    :recursive:
 
-   melodies_monet
-
+   driver
+   plots
+   stats
+   tutorial
+   util
 
+A typical workflow involves using an instance of :class:`driver.analysis`.
+See :doc:`the examples <examples/intro_examples>`.
 
 :ref:`genindex`
 

docs/appendix/machine-specific-install.rst

@@ -113,7 +113,7 @@ environment for running and developing MELODIES MONET.
 .. note::
    In the recent past, downloading a lot of dependent packages at once
    with conda on Hera led to stalling.
-   To overcome this challange, try installing packages individually::
+   To overcome this challenge, try installing packages individually::
   
         $ conda create --name melodies-monet python=3.9
         $ conda activate melodies-monet

docs/applications/forecasts.rst

@@ -12,9 +12,15 @@ CO, and NO\ :sub:`2` and AERONET AOD measurements.
 
 Check out the full analysis for each forecast `here <https://rapidrefresh.noaa.gov/RAPchemEPA/>`__.
 
+This includes a new feature developed by NOAA GSL summer student, Mackenzie Arnold,
+to `interactively view plots for individual surface sites online <https://rapidrefresh.noaa.gov/RAPchemEPAsites/>`__.
+
+The code to produce this analysis using MELODIES MONET is in the
+``examples/forecast_evaluation`` folder on GitHub.
+
 Example plots for ozone and PM\ :sub:`2.5` for the forecast on January 18th, 2022 
 are below.
 
 .. figure:: /_static/figures/OZONE_EPA_f00_rapchem.png
 
-.. figure:: /_static/figures/PM25_EPA_f00_rapchem.png
+.. figure:: /_static/figures/PM25_EPA_f00_rapchem.png

docs/background/supported_analyses.rst

@@ -0,0 +1,38 @@
+Supported Analyses
+==================
+
+Supported data analysis options in MELODIES MONET are explained below.
+
+Calculating Regulatory Metrics
+------------------------------
+
+MDA8 (8-hour daily maximum) ozone and 24 hour average PM\ :sub:`2.5` \ can be
+calculated within MELODIES MONET from hourly observational and model data. All plots
+except for the "spatial_overlay" plot and all stats will work with the regulatory
+metrics.
+
+The MDA8 ozone metric is calculated based on the following:
+  1) Local time is used
+  2) For each day, rolling 8-hour averages are calculated for each period with at least 6 hours of data
+  3) The MDA8 value is the highest of the 8-hour averages in a given day
+  4) The MDA8 value is only used for a given day if at least 18 of the 24 possible 8-hour averages are available
+
+As described in the EPA report on
+`Health Risk and Exposure Assessment for Ozone <https://www3.epa.gov/ttn/naaqs/standards/ozone/data/20140131healthrea4a.pdf>`__
+(page 8).
+
+The 24 hour average PM\ :sub:`2.5` \ metric is calculated based on the following:
+  1) Local time is used
+  2) The average PM\ :sub:`2.5` \ value is calculated over a given day
+  3) The PM\ :sub:`2.5` \ value is only used for a given day if at least 18 of the 24 possible hours are available
+
+In order to calculate the regulatory metric, add ``regulatory: True`` into the input
+YAML file under the "obs" section for each variable that you want to apply the calculation.
+Currently, this option only works for "OZONE" and "PM2.5" variables. There are separate
+plotting characteristics for the regulatory options ("ylabel_reg_plot", "vmin_reg_plot",
+"vmax_reg_plot", and "vdiff_reg_plot") that can also be specified in the input YAML file.
+An example input YAML file that calculates MDA8 ozone and 24 hr PM\ :sub:`2.5` \ is in
+``examples/yaml/control_rrfs_cmaq_airnow_reg.yaml``. These input YAML file options are
+also further described in the Appendix under :doc:`/appendix/yaml`.
+
+

docs/background/supported_datasets.rst

@@ -11,9 +11,9 @@ Supported Models
    * `CMAQ <https://www.epa.gov/cmaq/>`_
    * `WRF-Chem <https://ruc.noaa.gov/wrf/wrf-chem/>`_
    * `CESM/CAM-chem <https://www2.acom.ucar.edu/gcm/cam-chem>`_
-   * RRFS-CMAQ (under development)
-   * GSD-Chem (under development)
-
+   * `RRFS-CMAQ (under development) <https://github.com/ufs-community/ufs-srweather-app/wiki/Air-Quality-Modeling>`_ 
+   * `RAQMS <http://raqms-ops.ssec.wisc.edu/>`_
+   
 In general, processing requires input to be in netCDF format. For the above 
 models, scripts to configure the model data into a standard format for 
 MELODIES MONET are available. If input datasets are in netCDF format and  

docs/background/supported_plots.rst

@@ -28,7 +28,16 @@ Surface Evaluation
 .. figure:: /_static/figures/plot_grp3.spatial_bias.OZONE.2019-08-01_12.2019-08-11_12.all.CONUS.airnow_rrfs_diurnal_fires.png
 
    **Spatial Bias** - Difference plot of model - observations averaged over
-   the analysis window with options to specify the domain.
+   the analysis window with options to specify the domain. Defaults to average,
+   but users can also optionally plot percentiles.
+
+.. figure:: /_static/figures/plot_grp6.spatial_bias_exceedance.OZONE_reg.2019-08-01_13.2019-08-31_12.all.CONUS.airnow_cmaq_oper_exceedance.png
+
+   **Spatial Bias Exceedance** - Difference plot of model - observations for the number of
+   exceedances greater than the regulatory standard within the analysis window with options to specify
+   the domain. This only works for regulatory calculations (regulatory = True) for variables "OZONE" and "PM2.5" and units must be in ppbv or μg m\ :sup:`-3`\, respectively, after the ``unit_scale`` option in the control file is applied.
+   An exceedance occurs when MDA8 ozone is greater than 70 ppbv or 24 hour averaged PM\ :sub:`2.5` \ is
+   greater than 35 ug m\ :sup:`-3`\.
 
 .. figure:: /_static/figures/plot_grp4.spatial_overlay.OZONE.2019-08-01_12.2019-08-11_12.all.CONUS.airnow_rrfs_diurnal_fires.png
 
@@ -40,10 +49,7 @@ Surface Evaluation
    :scale: 25 %
 
    **BOX-plot** - BOX-plot comparing one or more model results with one
-   observation over the analysis window with options to specify the domain.	   
-
-Adding an option to plot regulatory metrics like MDA8 ozone is under 
-development.
+   observation over the analysis window with options to specify the domain.
 
 See the `Expand Surface Observations <https://github.com/NOAA-CSL/MELODIES-MONET/projects/7>`_ 
 project on GitHub to learn about current and future development.

docs/cli.rst

@@ -2,6 +2,30 @@
 Command Line Interface
 ======================
 
+Installing :mod:`melodies_monet` provides a command line interface (CLI):
+``melodies-monet``.
+
+For example, you can use the CLI to run control files without writing
+any Python code::
+
+    melodies-monet run control.yml
+
+**Subcommands**
+
+* |run|_ -- run a control file
+* |get-airnow|_ -- get AirNow data
+* |get-aeronet|_ -- get AERONET data
+
+.. |run| replace:: ``run``
+.. _run: #melodies-monet-run
+
+.. |get-airnow| replace:: ``get-airnow``
+.. _get-airnow: #melodies-monet-get-airnow
+
+.. |get-aeronet| replace:: ``get-aeronet``
+.. _get-aeronet: #melodies-monet-get-aeronet
+
+
 .. click:: melodies_monet._cli:_typer_click_object
    :prog: melodies-monet
    :nested: full

docs/conf.py

@@ -36,6 +36,7 @@
     'myst_nb',
     'sphinx_design',
     'sphinx_click',
+    'sphinx_togglebutton',
 ]
 
 extlinks = {

docs/develop/datasets.rst

@@ -5,17 +5,19 @@ Observations
 ------------
 
 Examples for how to read in data from Aeronet, AirNow, and Improve are in the
-``examples/process_obs`` folder of the code on GitHub. Use these examples
-as reference in order to add new surface observational datasets.
+``examples/process_obs`` folder in the MELODIES MONET repository
+`on GitHub <https://github.com/NOAA-CSL/MELODIES-MONET>`__.
+Use these examples as reference in order to add new surface observational datasets.
 
 Instructions for reading in aircraft and satellite observations are under development. 
 
 Models
 ------
 Examples for reading model datasets can be
-found in the monetio/models folder.
+found in the ``monetio/models`` folder in the MONETIO repository
+`on GitHub <https://github.com/noaa-oar-arl/monetio>`__.
 These include e.g., _cesm_fv_mm.py, _cmaq_mm.py, and _wrfchem_mm.py.
-While a part of the monetio repository,
-the MELODIES-MONET readers are designated with an _mm.
+While a part of the MONETIO repository,
+the private MELODIES MONET readers are designated with prefix ``_mm``.
 
 Support for additional models is also under developed.

docs/develop/developers_guide.rst

@@ -74,7 +74,7 @@ these instructions:
 
        $ conda create --name melodies-monet python=3.9
        $ conda activate melodies-monet
-       $ conda install -y -c conda-forge pyyaml monet monetio netcdf4 wrf-python click pooch jupyterlab
+       $ conda install -y -c conda-forge pyyaml monet monetio netcdf4 wrf-python typer rich pooch jupyterlab
 
 (b) Clone [#clone]_ and link the latest development versions of MONET and MONETIO from GitHub to
     your conda environment::
@@ -154,7 +154,7 @@ Either use the ``docs/environment-docs.yml`` file [#env]_
 from the MELODIES MONET repository,
 or add the following packages to your conda environment manually::
 
-    $ conda install -y -c conda-forge sphinx sphinx_rtd_theme myst-nb sphinx-design
+    $ conda install -y -c conda-forge sphinx sphinx_rtd_theme myst-nb sphinx-design sphinx-click sphinx-togglebutton typer
 
 The restructured text sources (rst) are located
 in the MELODIES-MONET/docs folders.

docs/develop/development_team.rst

@@ -15,19 +15,43 @@ NOAA GSL         Jordan Schnell
 NCAR ACOM        Louisa Emmons, David Fillmore        
 ===============  =============================
 
+Code Reviewers
+--------------
+
+================= =============== ========================================================
+Reviewer          GitHub Username Expertise
+================= =============== ========================================================
+Zachary Moon      zmoon           Docs, Connections to MONET and MONETIO, CI tests
+David Fillmore    dwfncar         Satellite obs, Plots, Stats, Connection to METplus
+Colin Harkins     colin-harkins   Docs, Aircraft obs, Plots, Stats
+Louisa Emmons     lkemmons        Docs, Aircraft obs, Plots, Stats
+Rebecca Schwantes rschwant        Docs, Surface and aircraft obs, Plots, Stats
+================= =============== ========================================================
+
 Development Team Members
 ------------------------
 
 Below is a list of people who have contributed to MELODIES MONET along with 
 their current and future development goals.
 
+**Mackenzie Arnold:**
+I’m a summer intern at CIRES/NOAA GSL working on model verification of smoke-forecast
+models. I plan to help review the code, work to improve the site-specific analysis, add
+in a new capability to interactively view plots for individual surface sites online,
+and help with various other aspects of development. 
+
 **Megan Bela:**
 I am a research scientist at CIRES/NOAA CSL working to improve process-level
 understanding and representations of fire emissions, plume rise, and chemistry
 in regional coupled chemistry-meteorology models. My plans include testing and
 expanding the capability of MELODIES MONET for evaluating simulations with
 research and operational models of fire impacts on air quality and weather.
 
+**Margaret Bruckner:**
+I am a graduate student at the University of Wisconsin-Madison. My development plans
+primarily focus on adding capabilities for comparison of satellite observations to model
+output and expanding data processing options. 
+
 **Rebecca Buchholz:**
 I am a Project Scientist at NCAR/ACOM helping with the development of MELODIES MONET.
 My development plans include incorporating satellite datasets such as MOPITT CO and

docs/environment-docs.yml

@@ -4,12 +4,14 @@ channels:
   - conda-forge
   - nodefaults
 dependencies:
-  - python=3.8
+  - python=3.9
   #
   # melodies_monet deps
   - monet
   - monetio
   - netcdf4
+  - numpy
+  - pandas<2
   #
   # Extras
   - pooch
@@ -27,6 +29,7 @@ dependencies:
   - sphinx-click
   - sphinx-design
   - sphinx_rtd_theme
+  - sphinx-togglebutton
   #
   - pip
   - pip:

docs/examples/airnow_camchem_se.ipynb

@@ -74,7 +74,7 @@
    "source": [
     "## Control File\n",
     "\n",
-    "Read in the required yaml control file that sets up all the definitons of what we want to pair and plot."
+    "Read in the required yaml control file that sets up all the definitions of what we want to pair and plot."
    ]
   },
   {