Skip to content

Commit

Permalink
Merge branch 'development' into metgrid_interpolator_bugfixes
Browse files Browse the repository at this point in the history
  • Loading branch information
wiersema1 committed Dec 20, 2024
2 parents 349a68a + b80c2b2 commit 3ca692a
Show file tree
Hide file tree
Showing 197 changed files with 1,731 additions and 982 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ jobs:
runs-on: ${{matrix.os}}
strategy:
matrix:
os: [ubuntu-latest]
os: [ubuntu-22.04]
include:
- os: ubuntu-latest
- os: ubuntu-22.04
install_deps: sudo apt-get install mpich libmpich-dev
comp: gnu
procs: $(nproc)
Expand Down
1 change: 1 addition & 0 deletions CMake/BuildERFExe.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ function(build_erf_lib erf_lib_name)
${SRC_DIR}/Utils/ERF_MomentumToVelocity.cpp
${SRC_DIR}/LinearSolvers/ERF_PoissonSolve.cpp
${SRC_DIR}/LinearSolvers/ERF_PoissonSolve_tb.cpp
${SRC_DIR}/LinearSolvers/ERF_PoissonWallDist.cpp
${SRC_DIR}/LinearSolvers/ERF_ComputeDivergence.cpp
${SRC_DIR}/LinearSolvers/ERF_SolveWithGMRES.cpp
${SRC_DIR}/LinearSolvers/ERF_SolveWithMLMG.cpp
Expand Down
2 changes: 1 addition & 1 deletion CMake/SetAmrexOptions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ set(AMReX_FORTRAN OFF)

set(AMReX_LINEAR_SOLVERS ON)
set(AMReX_LINEAR_SOLVERS_EM OFF)
set(AMReX_LINEAR_SOLVERS_INCFLO OFF)
set(AMReX_LINEAR_SOLVERS_INCFLO ON)

set(AMReX_PARTICLES OFF)
if(ERF_ENABLE_PARTICLES)
Expand Down
56 changes: 38 additions & 18 deletions Docs/sphinx_doc/BestPractices.rst
Original file line number Diff line number Diff line change
Expand Up @@ -45,41 +45,61 @@ Large-Eddy Simulations
* Time Integration

- Split timestepping offers some computational cost savings but still does
not allow you to run with an incompressible/anelastic time-step size.
- The acoustic CFL should be less than 0.5, with 4--6 fast timesteps
(substeps) according to WRF best practices.
not allow you to run with an incompressible/anelastic time-step size in
general.
- The acoustic CFL should conservatively be less than or equal to 0.5, with
4--6 fast timesteps (substeps) according to WRF best practices. If not
explicitly specified (through ``erf.fixed_mri_dt_ratio`` or
``erf.fixed_fast_dt``, the number of substeps in ERF is chosen based on the
same algorithm as WRF. If the user follows the recommendation that
dt [s] ~ 6 dx [km],
then 4 substeps will be used, giving an effective CFL of approximately 0.5.
This meets the stability criteria from Wicker & Skamarock 2002 that, for a
5th-order scheme, the CFL be less than 1.42/sqrt(3) = 0.820.

.. code-block:: python
erf.fixed_dt = 0.06 # slow timestep
# These are equivalent and result in a fixed fast timestep size
# if dx=10, speed of sound ~ 350 m/s
erf.fixed_fast_dt = 0.01 # ==> CFL~0.35
#erf.fixed_mri_dt_ratio = 6
# Alternatively, let ERF chose the fast timestep
#erf.cfl = 0.5
- We note that ERF LESs with up to 10 fast timesteps have successfully been
# if dx=10, speed of sound ~ 300 m/s
erf.fixed_mri_dt_ratio = 4
# or
#erf.fixed_fast_dt = 0.015 # ==> CFL~0.45
# or, let ERF chose the fast timestep
#erf.substepping_cfl = 0.5
- Following the WRF guidelines for dt is conservative. More aggressive time
integration--i.e., larger time steps with more substeps--is possible. We
note that ERF LESs with 10 or more fast timesteps have successfully been
run but your mileage may vary.


Single-Column Model
-------------------

* Currently, ERF does not have the ability to run a true single-column model
(SCM). The grid size in the lateral directions must have a minimum number of
cells. This will give comparable results, e.g.:
* An SCM is set up with a single cell in the lateral directions:

.. code-block:: python
geometry.prob_extent = 400 400 400
amr.n_cell = 2 2 64
amr.n_cell = 1 1 64
geometry.is_periodic = 1 1 0
When set up this way, the solution is not sensitive to horizontal problem
extent.

* An SCM was successfully run with third-order advection in the horizontal and
vertical.


2-D Cases
---------

* A 2-D planar domain can be configured as follows:

.. code-block:: python
geometry.prob_extent = 10000 100 1000
amr.n_cell = 100 1 20
geometry.is_periodic = 0 0 0
ylo.type = "SlipWall"
yhi.type = "SlipWall"
99 changes: 99 additions & 0 deletions Docs/sphinx_doc/CouplingToNoahMP.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
.. role:: cpp(code)
:language: c++

Coupling to Noah-MP
===================

Overview
--------

The NOAH Land Surface Model (LSM) is integrated with ERF to facilitate
interaction with the Noah-MP (Multi-Physics) land surface processes.

This documentation covers key components of this interface and its
associated data structures and routines, which are implemented in C++
and Fortran, and provides details on initialization and management of
data structures necessary for these processes.

Files Overview
--------------

- **Source/LandSurfaceModel/NOAH/ERF_NOAH.H**: Contains the declaration
of the NOAH class, which extends the NullSurf.

- **Source/LandSurfaceModel/NOAH/ERF_NOAH.cpp**: Implements the
initialization routine for the NOAH class.

- **Submodules/NOAH-MP/drivers/hrldas/NoahmpIO.H**: Defines the C++
`NoahmpIO_type` that is used to interface with Noah-MP implementations
following similar structure as the underlying Fortran interface
(https://dx.doi.org/10.5065/ew8g-yr95).

- **Submodules/NOAH-MP/drivers/hrldas/NoahmpIO.cpp**: Contains the
implementation of C++ routines interfacing with Fortran.

- **Submodules/NOAH-MP/drivers/hrldas/NoahmpIO_fi.F90**: Fortran module
responsible for managing mapping data between C++ and Fortran.

NOAH Class
----------

The NOAH class serves as the handler for initializing and managing the
data structures required for NOAH-MP operations. It inherits from the
`NullSurf` class. This class declares private variable `NoahmpIO_type
noahmpio`, that is passed to NoahMP routines similar to the Fortran
interface in the Noah-MP documentation
(https://dx.doi.org/10.5065/ew8g-yr95)

NoahmpIO_type Structure
-----------------------

This structure is key for handling the input and output operations for
NOAH-MP through C++ and Fortran interoperation. Contains various
variables for domain, memory, and tile configuration. Also, contains
arrays for geographic variables. At present this type exposes only a
select set of variables. More variables should be exposed as needed by
applications in ERF. The process of adding new variables is as follows:

#. In **Submodules/NOAH-MP/drivers/hrldas/NoahmpIO.H** add pointers to
the desired variable and set their initialization for
`NoahmpIO_type_fi` similar to implementation of `WSLAKEXY` and
`XLAT`.

#. In **Submodules/NOAH-MP/drivers/hrldas/NoahmpIO.H** declare objects
for Fortran-style multidimensional arrays for the same variables in
`NoahmpIO_type` similar to implemnation of `NoahArray2D<double> XLAT`
and `NoahArray2D<double> WSLAKEXY`.

#. In **Submodules/NOAH-MP/drivers/hrldas/NoahmpIO.cpp** cast the
pointers from `NoahmpIO_type_fi` to multidimensional arrays in
`NoahmpIO_type` within the implementation of `void
NoahmpIOVarInitDefault(NoahmpIO_type* noahmpio)`.

Fortran Interoperability
------------------------

The connection between C++ and Fortran is managed through `NoahmpIO_fi`.
This module contains a mirroring of the C++ structure for NOAH-MP
input-output operations.

The following functions are used to operate on the `NoahmpIO_type` and
interface with their respective Fortran implementations:

- `void NoahmpIOVarInitDefault(NoahmpIO_type* noahmpio)`: Initializes
default variables of `NoahmpIO_type`. Create C pointer for Fortran
data.

- `void NoahmpInitMain(NoahmpIO_type* noahmpio)`: Main initialization
function for the NOAH-MP operations in C++.

Usage
-----

To use the NOAH class and associated functions, ensure the correct
initialization sequence is followed within the simulation setup. The
interplay between C++ and Fortran necessitates careful memory and data
handling, which is crucial for ensuring performance and correctness in
simulations. The interface is designed to mimic the Fortran interface
from documentation(https://dx.doi.org/10.5065/ew8g-yr95), therefore
similar practices should be followed.
43 changes: 31 additions & 12 deletions Docs/sphinx_doc/Inputs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -906,7 +906,7 @@ List of Parameters
| | viscosity and | "Constant", or | |
| | diffusivity? | "ConstantAlpha" | |
+----------------------------------+--------------------+---------------------+--------------+
| **erf.dynamicViscosity** | Viscous coeff. if | Real | 0.0 |
| **erf.dynamic_viscosity** | Viscous coeff. if | Real | 0.0 |
| | DNS | | |
+----------------------------------+--------------------+---------------------+--------------+
| **erf.Cs** | Constant | Real | 0.0 |
Expand All @@ -918,13 +918,11 @@ List of Parameters
| **erf.Sc_t** | Turbulent Schmidt | Real | 1.0 |
| | Number | | |
+----------------------------------+--------------------+---------------------+--------------+
| **erf.use_NumDiff** | Use 6th order | "true", | "false" |
| | numerical diffusion| "false" | |
| | | | |
+----------------------------------+--------------------+---------------------+--------------+
| **erf.NumDiffCoeff** | Coefficient for | Real | 0.0 |
| | 6th order | [0.0, 1.0] | |
| | numerical diffusion| | |
| **erf.num_diff_coeff** | Coefficient for | Real | 0.0 |
| | 6th-order | [0.0, 1.0] | |
| | numerical | | |
| | diffusion, set to 0| | |
| | to disable | | |
+----------------------------------+--------------------+---------------------+--------------+

Note: in the equations for the evolution of momentum, potential temperature and advected scalars, the
Expand Down Expand Up @@ -1089,11 +1087,32 @@ List of Parameters
| **erf.coriolis_3d** | Include z component in | true / false | true |
| | the Coriolis forcing | | |
+-------------------------------------+------------------------+-------------------+---------------------+
| **erf.use_rayleigh_damping** | Include explicit | true / false | false |
| | Rayleigh damping | | |
| **erf.rayleigh_damp_U** | Include explicit | true / false | false |
| | Rayleigh damping in | | |
| | the x-momentum equation| | |
+-------------------------------------+------------------------+-------------------+---------------------+
| **erf.rayleigh_damp_V** | Include explicit | true / false | false |
| | Rayleigh damping in | | |
| | the y-momentum equation| | |
+-------------------------------------+------------------------+-------------------+---------------------+
| **erf.rayleigh_damp_W** | Include explicit | true / false | false |
| | Rayleigh damping in | | |
| | the z-momentum equation| | |
+-------------------------------------+------------------------+-------------------+---------------------+
| **erf.nudging_from_input_sounding** | Include explicit | true / false | false |
| | Rayleigh damping | | |
| **erf.rayleigh_damp_T** | Include explicit | true / false | false |
| | Rayleigh damping in | | |
| | the potential | | |
| | temperature equation | | |
+-------------------------------------+------------------------+-------------------+---------------------+
| **erf.rayleigh_dampcoef** | Rayleigh damping | Real | 0.2 |
| | coefficient, an inverse| | |
| | timescale | | |
+-------------------------------------+------------------------+-------------------+---------------------+
| **erf.nudging_from_input_sounding** | Add momentum source | true / false | false |
| | terms to nudge the | | |
| | solution towards the | | |
| | initial sounding | | |
| | profile | | |
+-------------------------------------+------------------------+-------------------+---------------------+
| **erf.input_sounding_file** | Name(s) of the | String(s) | input_sounding_file |
| | input sounding file(s) | | |
Expand Down
23 changes: 13 additions & 10 deletions Docs/sphinx_doc/Plotfiles.rst
Original file line number Diff line number Diff line change
Expand Up @@ -187,19 +187,13 @@ Output Options
| | |
| | |
+-----------------------------+------------------+
| **KE** | Kinetic energy |
| | |
| | |
+-----------------------------+------------------+
| **QKE** | Turbulent |
| **KE** | SGS turbulent |
| | kinetic energy |
| | * 2 |
+-----------------------------+------------------+
| **rhoKE** | Density * KE |
| | |
| | (from Deardorff |
| | or MYNN) |
| | |
+-----------------------------+------------------+
| **rhoQKE** | Density * QKE |
| **rhoKE** | Density * KE |
| | |
| | |
+-----------------------------+------------------+
Expand Down Expand Up @@ -263,13 +257,22 @@ Output Options
| | mass points |
| | |
+-----------------------------+------------------+
| **nut** | Eddy viscosity, |
| | nu_t |
+-----------------------------+------------------+
| **Kmv** | Vertical |
| | Eddy Diffusivity |
| | of Momentum |
+-----------------------------+------------------+
| **Kmh** | Horizontal |
| | Eddy Diffusivity |
| | of Momentum |
| | (Note: For LES, |
| | this is the |
| | _dynamic_ eddy |
| | viscosity, mu_t |
| | = rho * nu_t |
| | and Kmh==Kmv) |
+-----------------------------+------------------+
| **Khv** | Vertical |
| | Eddy Diffusivity |
Expand Down
1 change: 1 addition & 0 deletions Docs/sphinx_doc/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ In addition to this documentation, there is API documentation for ERF generated
CouplingToAMRWind.rst
CouplingToWW3.rst
CouplingToNoahMP.rst
.. toctree::
:caption: ERF vs WRF
Expand Down
3 changes: 2 additions & 1 deletion Exec/ABL/ERF_Prob.H
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ struct ProbParm : ProbParmDefaults {
amrex::Real rho_0 = 0.0;
amrex::Real T_0 = 0.0;
amrex::Real A_0 = 1.0;
amrex::Real KE_0 = 0.1;
amrex::Real KE_0 = 0.1; // initialize to calculated rho_hse times input KE
amrex::Real rhoKE_0 = -1; // alternatively, initialize to specified (rho*KE)

amrex::Real KE_decay_height = -1;
amrex::Real KE_decay_order = 1;
Expand Down
7 changes: 6 additions & 1 deletion Exec/ABL/ERF_Prob.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Problem::Problem(const amrex::Real* problo, const amrex::Real* probhi)
pp.query("T_0", parms.T_0);
pp.query("A_0", parms.A_0);
pp.query("KE_0", parms.KE_0);
pp.query("rhoKE_0", parms.rhoKE_0);
pp.query("KE_decay_height", parms.KE_decay_height);
pp.query("KE_decay_order", parms.KE_decay_order);

Expand Down Expand Up @@ -126,7 +127,11 @@ Problem::init_custom_pert(
// Set an initial value for SGS KE
if (state_pert.nComp() > RhoKE_comp) {
// Deardorff
state_pert(i, j, k, RhoKE_comp) = r_hse(i,j,k) * parms_d.KE_0;
if (parms_d.rhoKE_0 > 0) {
state_pert(i, j, k, RhoKE_comp) = parms_d.rhoKE_0;
} else {
state_pert(i, j, k, RhoKE_comp) = r_hse(i,j,k) * parms_d.KE_0;
}
if (parms_d.KE_decay_height > 0) {
// scale initial SGS kinetic energy with height
state_pert(i, j, k, RhoKE_comp) *= max(
Expand Down
2 changes: 0 additions & 2 deletions Exec/ABL/inputs.read
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,6 @@ erf.plot_int_1 = 100 # number of timesteps between plotfiles
erf.plot_vars_1 = density rhoadv_0 x_velocity y_velocity z_velocity pressure temp theta

# SOLVER CHOICE
erf.alpha_T = 0.0
erf.alpha_C = 1.0
erf.use_gravity = false

erf.molec_diff_type = "None"
Expand Down
2 changes: 0 additions & 2 deletions Exec/ABL/inputs.write
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,6 @@ erf.plot_int_1 = 10 # number of timesteps between plotfiles
erf.plot_vars_1 = density rhoadv_0 x_velocity y_velocity z_velocity pressure temp theta

# SOLVER CHOICE
erf.alpha_T = 0.0
erf.alpha_C = 1.0
erf.use_gravity = false

erf.molec_diff_type = "None"
Expand Down
2 changes: 0 additions & 2 deletions Exec/ABL/inputs_DataSampler
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,6 @@ erf.plot_int_1 = 10 # number of timesteps between plotfiles
erf.plot_vars_1 = density rhoadv_0 x_velocity y_velocity z_velocity pressure temp theta

# SOLVER CHOICE
erf.alpha_T = 0.0
erf.alpha_C = 1.0
erf.use_gravity = false

erf.molec_diff_type = "None"
Expand Down
Loading

0 comments on commit 3ca692a

Please sign in to comment.