refactor(createpackages): use jinja for mf6 module code generation

.docs/md/generate_classes.md

@@ -10,7 +10,9 @@
 
 MODFLOW 6 input continues to evolve as new models, packages, and options are developed, updated, and supported. All MODFLOW 6 input is described by DFN (definition) files, which are simple text files that describe the blocks and keywords in each input file. These definition files are used to build the input and output guide for MODFLOW 6. These definition files are also used to automatically generate FloPy classes for creating, reading and writing MODFLOW 6 models, packages, and options.  FloPy and MODFLOW 6 are kept in sync by these DFN (definition) files, and therefore, it may be necessary for a user to update FloPy using a custom set of definition files, or a set of definition files from a previous release.
 
-The FloPy classes for MODFLOW 6 are largely generated by a utility which converts DFN files in a modflow6 repository on GitHub or on the local machine into Python source files in your local FloPy install. For instance (output much abbreviated):
+The FloPy classes for MODFLOW 6 are largely generated by a utility which converts DFN files in a modflow6 repository on GitHub or on the local machine into Python source files in your local FloPy install.
+
+**Note**: to use this functionality, the `codegen` optional dependency group must be installed.
 
 ```bash
 $ python -m flopy.mf6.utils.generate_classes

.github/workflows/commit.yml

@@ -104,7 +104,7 @@ jobs:
         run: |
           pip install --upgrade pip
           pip install .
-          pip install ".[test, optional]"
+          pip install ".[dev]"
 
       - name: Install Modflow executables
         uses: modflowpy/install-modflow-action@v1
@@ -159,7 +159,9 @@ jobs:
             powershell
 
       - name: Install FloPy
-        run: pip install .
+        run: |
+          pip install .
+          pip install ".[dev]"
 
       - name: Install Modflow-related executables
         uses: modflowpy/install-modflow-action@v1

.github/workflows/examples.yml

@@ -34,7 +34,9 @@ jobs:
             powershell
 
       - name: Install FloPy
-        run: pip install .
+        run: |
+          pip install .
+          pip install ".[codegen]"
 
       - name: OpenGL workaround on Linux
         if: runner.os == 'Linux'

.github/workflows/mf6.yml

@@ -37,7 +37,7 @@ jobs:
         pip install https://github.com/modflowpy/pymake/zipball/master
         pip install https://github.com/Deltares/xmipy/zipball/develop
         pip install https://github.com/MODFLOW-USGS/modflowapi/zipball/develop
-        pip install .[test,optional]
+        pip install .[dev]
         pip install meson ninja
 
     - name: Setup GNU Fortran
@@ -120,7 +120,7 @@ jobs:
         pip install https://github.com/modflowpy/pymake/zipball/master
         pip install https://github.com/Deltares/xmipy/zipball/develop
         pip install https://github.com/MODFLOW-USGS/modflowapi/zipball/develop
-        pip install .[test,optional]
+        pip install .[dev]
         pip install meson ninja
         pip install -r modflow6-examples/etc/requirements.pip.txt
 

.github/workflows/rtd.yml

@@ -17,7 +17,7 @@ concurrency:
   cancel-in-progress: true
 jobs:
   set_options:
-    name: Set release options
+    name: Set options
     runs-on: ubuntu-22.04
     outputs:
       ref: ${{ steps.set_ref.outputs.ref }}

.gitignore

@@ -103,4 +103,7 @@ app
 **.DS_Store
 
 # DFN backups
-flopy/mf6/data/dfn_backup/
+flopy/mf6/data/dfn_backup/
+
+# DFN TOML dir
+flopy/mf6/data/toml/

autotest/test_codegen.py

@@ -0,0 +1,28 @@
+import pytest
+
+from autotest.conftest import get_project_root_path
+from flopy.mf6.utils.codegen import make_all
+from flopy.mf6.utils.codegen.dfn import Dfn
+
+PROJ_ROOT = get_project_root_path()
+MF6_PATH = PROJ_ROOT / "flopy" / "mf6"
+DFN_PATH = MF6_PATH / "data" / "dfn"
+DFN_NAMES = [
+    dfn.stem for dfn in DFN_PATH.glob("*.dfn") if dfn.stem not in ["common", "flopy"]
+]
+
+
+@pytest.mark.parametrize("dfn_name", DFN_NAMES)
+def test_dfn_load(dfn_name):
+    with (
+        open(DFN_PATH / "common.dfn", "r") as common_file,
+        open(DFN_PATH / f"{dfn_name}.dfn", "r") as dfn_file,
+    ):
+        name = Dfn.Name.parse(dfn_name)
+        common, _ = Dfn._load_v1_flat(common_file)
+        Dfn.load(dfn_file, name=name, common=common)
+
+
+def test_make_all(function_tmpdir):
+    make_all(DFN_PATH, function_tmpdir, verbose=True)
+    assert any(function_tmpdir.glob("*.py"))

autotest/test_generate_classes.py

@@ -94,7 +94,7 @@ def test_generate_classes_from_github_refs(
     print(f"Using temp venv at {venv} to test class generation from {ref}")
 
     # install flopy and dependencies
-    deps = [str(project_root_path), "modflow-devtools"]
+    deps = [str(project_root_path), "modflow-devtools", "Jinja2", "boltons"]
     for dep in deps:
         out, err, ret = run_cmd(str(pip), "install", dep, verbose=True)
         assert not ret, out + err

docs/mf6_dev_guide.md

@@ -1,47 +1,91 @@
-Introduction
------------------------------------------------
+# Developing FloPy for MF6
 
-This file provides an overview of how FloPy for MODFLOW 6 (FPMF6) works under the hood and is intended for anyone who wants to add a new package, new model type, or new features to this library.  FloPy library files that support MODFLOW 6 can be found in the flopy/mf6 folder and sub-folders. 
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 
-Package Meta-Data and Package Files
------------------------------------------------
+- [Introduction](#introduction)
+- [Code generation](#code-generation)
+- [Input specification](#input-specification)
 
-FPMF6 uses meta-data files located in flopy/mf6/data/dfn to define the model and package types supported by MODFLOW 6.  When additional model and package types are added to MODFLOW 6, additional meta-data files can be added to this folder and flopy/mf6/utils/createpackages.py can be run to add new packages to the FloPy library.  createpackages.py uses flopy/mf6/data/mfstructure.py to read meta-data files (*.dfn) and use that meta-data to create the package files found in flopy/mf6/modflow (do not directly modify any of the files in this folder, they are all automatically generated).  The automatically generated package files contain an interface for accessing package data and data documentation generated from the meta-data files.  Additionally, meta-data describing package data types and shapes is stored in the dfn attribute.  flopy/mf6/data/mfstructure.py can load structure information using the dfn attribute (instead of loading it from the meta-data files).  This allows for flopy to be installed without the dfn files.
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
-All meta-data can be accessed from the flopy.mf6.data.mfstructure.MFStructure class.  This is a singleton class, meaning only one instance of this class can be created.  The class contains a sim_struct attribute (which is a flopy.mf6.data.mfstructure.MFSimulationStructure object) which contains all of the meta-data for all package files.  Meta-data is stored in a structured format. MFSimulationStructure contains MFModelStructure and MFInputFileStructure objects, which contain the meta-data for each model type and each "simulation-level" package (tdis, ims, ...).  MFModelStructure contains model specific meta-data and a MFInputFileStructure object for each package in that model.  MFInputFileStructure contains package specific meta-data and a MFBlockStructure object for each block contained in the package file.  MFBlockStructure contains block specific meta-data and a MFDataStructure object for each data structure defined in the block, and MFDataStructure contains data structure specific meta-data and a MFDataItemStructure object for each data item contained in the data structure.  Data structures define the structure of data that is naturally grouped together, for example, the data in a numpy recarray.  Data item structures define the structure of specific pieces of data, for example, a single column of a numpy recarray.  The meta-data defined in these classes provides all the information FloPy needs to read and write MODFLOW 6 package and name files, create the Flopy interface, and check the data for various constraints.
+## Introduction
 
+This file provides an overview of how FloPy's MODFLOW 6 module `flopy.mf6` works under the hood. It is intended for FloPy maintainers, as well as anyone who wants to add a new package, new model, or new features to this library.
 
-***
-MFStructure --+ MFSimulationStructure --+ MFModelStructure --+ MFInputFileStructure --+ MFBlockStructure --+ MFDataStructure --+ MFDataItemStructure
+## Code generation
 
-Figure 1: FPMF6 generic data structure classes.  Lines connecting classes show a relationship defined between the two connected classes.  A "*" next to the class means that the  class is a sub-class of the connected class.  A "+" next to the class means that the class is contained within the connected class.
-***
+MODFLOW 6 describes its input specification with definition files. These are currently a custom text-based format. Definition files have suffix `.dfn` by convention.
 
-Package and Data Base Classes
------------------------------------------------
+We plan to move soon to TOML definition files. More on this below.
 
-The package and data classes are related as shown below in figure 2.  On the top of the figure 2 is the MFPackage class, which is the base class for all packages.  MFPackage contains generic methods for building data objects and reading and writing the package to a file.  MFPackage contains a MFInputFileStructure object that defines how the data is structured in the package file.  MFPackage also contains a dictionary of blocks (MFBlock).  The MFBlock class is a generic class used to represent a block within a package.  MFBlock contains a MFBlockStructure object that defines how the data in the block is structured.  MFBlock also contains a dictionary of data objects (subclasses of MFData) contained in the block and a list of block headers (MFBlockHeader) for that block.  Block headers contain the block's name and optionally data items (eg. iprn).
+Definition files describe components (e.g. simulations, models, packages) supported by MODFLOW 6, and are used to generate both source code and documentation.
+
+FloPy has two scripts that can be used to generate a MODFLOW 6 compatibility layer:
+
+- `flopy/mf6/utils/createpackages.py`: assumes definition files are in `flopy/mf6/data/dfn`
+- `flopy/mf6/utils/generate_classes.py`: downloads DFNs then runs `createpackages.py`
+
+The latter is typically used with e.g. `python -m flopy.mf6.utils.generate_classes --ref develop`.
+
+Generated files are created in `flopy/mf6/modflow/` and contain interface classes, one file/class per input component. These can be used to initialize and access model/package data as well as the input specification itself.
+
+**Note**: Code generation requires a few extra dependencies, grouped in the `codegen` optional dependency group: `Jinja2`, `boltons`, `tomlkit` and `modflow-devtools`.
 
+**Note**: Code generation scripts previously used `flopy/mf6/data/mfstructure.py` to read and represent definition files, and wrote Python by hand. They now use the `flopy.mf6.utils.codegen` module, which uses Jinja2.
 
-***
-MFPackage --+ MFBlock --+ MFData
+**Note**: Each component's input definition is currently reproduced almost verbatim in the `dfn` class attribute. Currently, `flopy/mf6/data/mfstructure.py` is used to introspect the input specification using the `dfn` attribute. This can eventually be removed in favor of direct introspection.
 
-MFPackage --+ MFInputFileStructure
+The `flopy.mf6.utils.codegen` module is small and meant to be easy to iterate on. Its purpose is to load and convert input definition files to Python source code by way of an intermediate representation.
 
-MFBlock --+ MFBlockStructure
+As such, there are 2 abstractions: `Dfn` and `Context`. A `Dfn` corresponds to a definition file, and knows how to load itself from one. A `Context` corresponds to a Python source file, which it is fed to a Jinja template to create. ('Context' is a term borrowed from Jinja.) `Dfn` and `Context` typically map 1-1, but can be 1-many (e.g. a model definition file yields a model class and namefile package class).
 
-MFData --+ MFDataStructure
+Both `Dfn` and `Context` are structured representations of an input component/block/variable hierarchy. For now, we have to infer this structure from a flat representation in the definition file. This is a bit like [object-relational impedance mismatch](https://en.wikipedia.org/wiki/Object%E2%80%93relational_impedance_mismatch) and seriously complicates the `Dfn` load routines, but happily this is temporary &mdash; once we move to TOML and define input components in structured form, we can simply load the nested variable hierarchy directly from the definition file.
 
-MFData --* MFArray --* MFTransientArray
+For now we use a data structure from the `boltons` library to maintain an unstructured (flat) map of variables before structural parsing, where variables can have duplicate names.
 
-MFData --* MFList --* MFTransientList
+Some quirks of the legacy framework are handled in a "shim" of Jinja filters that transform the template context, as well as some macros. These can ideally be removed as refactoring goes on. The templates should also get simpler over time.
 
-MFData --* MFScalar --* MFTransientScalar
+## Input specification
+
+**Note**: the following describes the legacy input specification mechanism.
+
+The `flopy.mf6.data.mfstructure.MFStructure` class represents an input specification. The class is a singleton, meaning only one instance of this class can be created.  The class contains a sim_struct attribute (which is a flopy.mf6.data.mfstructure.MFSimulationStructure object) which contains all of the meta-data for all package files.  Meta-data is stored in a structured format. MFSimulationStructure contains MFModelStructure and MFInputFileStructure objects, which contain the meta-data for each model type and each "simulation-level" package (tdis, ims, ...).  MFModelStructure contains model specific meta-data and a MFInputFileStructure object for each package in that model.  MFInputFileStructure contains package specific meta-data and a MFBlockStructure object for each block contained in the package file.  MFBlockStructure contains block specific meta-data and a MFDataStructure object for each data structure defined in the block, and MFDataStructure contains data structure specific meta-data and a MFDataItemStructure object for each data item contained in the data structure.  Data structures define the structure of data that is naturally grouped together, for example, the data in a numpy recarray.  Data item structures define the structure of specific pieces of data, for example, a single column of a numpy recarray.  The meta-data defined in these classes provides all the information FloPy needs to read and write MODFLOW 6 package and name files, create the Flopy interface, and check the data for various constraints.
+
+```mermaid
+classDiagram
+    MFStructure *-- "1" MFSimulationStructure : has
+    MFSimulationStructure *-- "1+" MFModelStructure : has
+    MFModelStructure *-- "1" MFInputFileStructure : has
+    MFInputFileStructure *-- "1+" MFBlockStructure : has
+    MFBlockStructure *-- "1+" MFDataStructure : has
+    MFDataStructure *-- "1+" MFDataItemStructure : has
+```
+
+Figure 1: Generic data structure hierarchy.  Connections show composition relationships.
+
+The package and data classes are related as shown below in figure 2.  On the top of the figure 2 is the MFPackage class, which is the base class for all packages.  MFPackage contains generic methods for building data objects and reading and writing the package to a file.  MFPackage contains a MFInputFileStructure object that defines how the data is structured in the package file.  MFPackage also contains a dictionary of blocks (MFBlock).  The MFBlock class is a generic class used to represent a block within a package.  MFBlock contains a MFBlockStructure object that defines how the data in the block is structured.  MFBlock also contains a dictionary of data objects (subclasses of MFData) contained in the block and a list of block headers (MFBlockHeader) for that block.  Block headers contain the block's name and optionally data items (eg. iprn).
 
-MFTransientData --* MFTransientArray, MFTransientList, MFTransientScalar
+```mermaid
+classDiagram
+
+MFPackage *-- "1+" MFBlock : has
+MFBlock *-- "1+" MFData : has
+MFPackage *-- "1" MFInputFileStructure : has
+MFBlock *-- "1" MFBlockStructure : has
+MFData *-- "1" MFDataStructure : has
+MFData <|-- MFArray
+MFArray <|-- MFTransientArray
+MFData <|-- MFList
+MFList <|-- MFTransientList
+MFData <|-- MFScalar
+MFScalar <|-- MFTransientScalar
+MFTransientData <|-- MFTransientArray
+MFTransientData <|-- MFTransientList
+MFTransientData <|-- MFTransientScalar
+```
 
 Figure 2:  FPMF6 package and data classes.  Lines connecting classes show a relationship defined between the two connected classes.  A "*" next to the class means that the  class is a sub-class of the connected class.  A "+" next to the class means that the class is contained within the connected class.
-***
 
 There are three main types of data, MFList, MFArray, and MFScalar data.  All three of these data types are derived from the MFData abstract base class.  MFList data is the type of data stored in a spreadsheet with different column headings.  For example, the data describing a flow barrier are of type MFList.  MFList data is stored in numpy recarrays.  MFArray data is data of a single type (eg. all integer values).  For example, the model's HK values are of type MFArray.  MFArrays are stored in numpy ndarrays.  MFScalar data is a single data item.  Most MFScalar data are options.  All MFData subclasses contain an MFDataStructure object that defines the expected structure and types of the data.
 

etc/environment.yml

@@ -10,6 +10,13 @@ dependencies:
   - matplotlib>=1.4.0
   - pandas>=2.0.0
 
+  # codegen
+  - boltons>=1.0
+  - Jinja2>=3.0
+  - pip:
+      - git+https://github.com/MODFLOW-USGS/modflow-devtools.git
+  - tomlkit
+
   # lint
   - cffconvert
   - ruff
@@ -21,8 +28,6 @@ dependencies:
   - jupyter
   - jupyter_client>=8.4.0
   - jupytext
-  - pip:
-      - git+https://github.com/MODFLOW-USGS/modflow-devtools.git
   - pytest!=8.1.0
   - pytest-benchmark
   - pytest-cov

flopy/mf6/data/dfn/utl-tas.dfn

@@ -19,6 +19,7 @@ type keyword
 shape
 reader urword
 optional false
+in_record true
 longname
 description xxx
 
@@ -29,6 +30,7 @@ shape any1d
 tagged false
 reader urword
 optional false
+in_record true
 longname
 description Name by which a package references a particular time-array series. The name must be unique among all time-array series used in a package.
 
@@ -48,6 +50,7 @@ type keyword
 shape
 reader urword
 optional false
+in_record true
 longname
 description  xxx
 
@@ -59,6 +62,7 @@ shape
 tagged false
 reader urword
 optional false
+in_record true
 longname
 description Interpolation method, which is either STEPWISE or LINEAR.
 
@@ -78,6 +82,7 @@ type keyword
 shape
 reader urword
 optional false
+in_record true
 longname
 description  xxx
 
@@ -88,6 +93,7 @@ shape time_series_name
 tagged false
 reader urword
 optional false
+in_record true
 longname
 description Scale factor, which will multiply all array values in time series. SFAC is an optional attribute; if omitted, SFAC = 1.0.