Merge pull request #398 from SDXorg/dev-3.9.0

v3.9.0

README.md

@@ -33,7 +33,7 @@ For standard methods for data analysis with SD models, see the  [PySD Cookbook](
 
 ## Why create a new SD simulation engine?
 
-There are a number of great SD programs out there ([Vensim](http://vensim.com/), [iThink](http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx), [AnyLogic](http://www.anylogic.com/system-dynamics), [Insight Maker](http://insightmaker.com/), and [others](http://en.wikipedia.org/wiki/List_of_system_dynamics_software)). In order not to waste our effort, or fall victim to the [Not-Invented-Here](http://en.wikipedia.org/wiki/Not_invented_here) fallacy, we should have a very good reason for starting a new project.
+There are a number of great SD programs out there ([Vensim](http://vensim.com/), [iThink](http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx), [AnyLogic](http://www.anylogic.com/system-dynamics), [Insight Maker](http://insightmaker.com/), and [others](https://en.wikipedia.org/wiki/Comparison_of_system_dynamics_software)). In order not to waste our effort, or fall victim to the [Not-Invented-Here](http://en.wikipedia.org/wiki/Not_invented_here) fallacy, we should have a very good reason for starting a new project.
 
 That reason is this: There is a whole world of computational tools being developed in the larger data science community. **System dynamicists should directly use the tools that other people are building, instead of replicating their functionality in SD specific software.** The best way to do this is to bring specific SD functionality to the domain where those other tools are being developed.
 

docs/command_line_usage.rst

@@ -58,7 +58,15 @@ In order to split the Vensim model views in different files, as explained in :do
 
 .. code-block:: text
 
-    python -m pysd --split-views many_views_model.mdl
+    python -m pysd many_views_model.mdl --split-views
+
+The previous code will put each model view in a separate Python module. Additionally, if the names of the views include the concepts of subsubmodules (e.g., ENERGY-transformation.efficiency_improvement), the *--subview-sep* (subview separators) argument may be used to further classify the model equations:
+
+.. code-block:: text
+
+    python -m pysd many_views_and_subviews_model.mdl --split-views --subview-sep - .
+
+Note that passing any positional argument right after the *--subview-sep* argument will raise an error, so it is recommended to pass this argument as the last one.
 
 
 Outputting various run information
@@ -102,7 +110,7 @@ modified using the *-I/--initial_time*, *-F/--final-time*, *-T/--time-step* and
 
 .. code-block:: text
 
-    python -m pysd -I 2005 --final-time=2010 --time-step=1 Teacup.mdl
+    python -m pysd -I=2005 --final-time=2010 --time-step=1 Teacup.mdl
 
 will set the initial time to 2005, the final time to 2010 and the time step to 1.
 

docs/development/adding_functions.rst

@@ -1,6 +1,6 @@
 Adding new functions
 ====================
-In this section you may found some helpful examples for adding a new function to the PySD Python builder. Before starting adding any new feature or fuction, please, make sure that no one is working on it. Search if any open issue exists with the feature you want to work on or open a new one if it does not exist. Then, claim that you are working on it.
+In this section you may find some helpful examples for adding a new function to the PySD Python builder. Before starting adding any new feature or function, please, make sure that no one is working on it. Search if any open issue exists with the feature you want to work on or open a new one if it does not exist. Then, claim that you are working on it.
 
 Adding a hardcoded function
 ---------------------------

docs/structure/structure_index.rst

@@ -33,7 +33,7 @@ While something like :py:data:`1<5 and 5>3`::
 
    LogicStructure(operators=[':AND:'], arguments=(LogicStructure(operators=['<'], arguments=(1, 5)), LogicStructure(operators=['>'], arguments=(5, 3))))
 
-The parenthesis also affects same order operatos, for example :py:data:`1+2-3` is translated to::
+The parenthesis also affects same order operators, for example :py:data:`1+2-3` is translated to::
 
    ArithmeticStructure(operators=['+', '-'], arguments=(1, 2, 3))
 

docs/whats_new.rst

@@ -1,5 +1,38 @@
 What's New
 ==========
+v3.9.0 (2022/12/15)
+-------------------
+
+New Features
+~~~~~~~~~~~~
+- Parses and ignores reality check functions during translation of Vensim models. (`@rogersamso <https://github.com/rogersamso>`_)
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+Deprecations
+~~~~~~~~~~~~
+
+Bug fixes
+~~~~~~~~~
+- Fix issue with the classification of variables in modules and submodules (:issue:`388`). When a model had a view with 3 sublevels (e.g. energy-transformation.losses) but another view was defined with only two of them (e.g. energy-transformation), the variables in the second view were placed in the main model file. Now, if this happens, the variables in the second view will be placed in a main.py file (i.e. energy/transformation/main.py). (`@rogersamso <https://github.com/rogersamso>`_)
+- Fix bug on the CLI when passing a hyphen as first value to the *--subview-sep* argument (:issue:`388`). (`@rogersamso <https://github.com/rogersamso>`_)
+- Fix bug on the CLI when parsing initial conditions (:issue:`395`). (`@rogersamso <https://github.com/rogersamso>`_)
+
+Documentation
+~~~~~~~~~~~~~
+- The `Splitting Vensim views in different files` section in :doc:`command_line_usage` has been updated to include an example of the usage of the *--subview-sep* CLI argument. (`@rogersamso <https://github.com/rogersamso>`_)
+
+Performance
+~~~~~~~~~~~
+
+Internal Changes
+~~~~~~~~~~~~~~~~
+- The :py:meth:`_merge_nested_dicts` method from the :py:class:`pysd.translators.vensim.vensim_file.VensimFile` class has been made a static method, as it does not need to access any attribute of the instance, and it does facilitate unit testing. (`@rogersamso <https://github.com/rogersamso>`_)
+- The `pysd/translators/vensim/parsing_grammars/element_object.peg` grammar has been modified to be able to parse reality check elements. (`@rogersamso <https://github.com/rogersamso>`_)
+- :py:class:`pysd.translators.vensim.vensim_element.Constraint`  and :py:class:`pysd.translators.vensim.vensim_element.TestInputs` classes have been added, which inherit from the also newly created :py:class:`pysd.translators.vensim.vensim_element.GenericComponent`, which include the :py:meth:`parse` and :py:meth:`get_abstract_component` methods. (`@rogersamso <https://github.com/rogersamso>`_ and `@enekomartinmartinez <https://github.com/enekomartinmartinez>`_)
+- The :py:class:`pysd.translators.structures.abstract_model.AbstractSection` class now has two extra attributes (:py:data:`constraints` and :py:data:`input_tests`), which hold the :py:class:`pysd.translators.structures.abstract_model.AbstractConstraint` and :py:class:`pysd.translators.structures.abstract_model.AbstractTestInputs` objects. (`@rogersamso <https://github.com/rogersamso>`_)
+
 v3.8.0 (2022/11/03)
 -------------------
 

pysd/_version.py

@@ -1 +1 @@
-__version__ = "3.8.0"
+__version__ = "3.9.0"

pysd/builders/python/python_expressions_builder.py

@@ -130,7 +130,7 @@ def reshape(self, subscripts: SubscriptManager,
     def lower_order(self, new_order: int) -> None:
         """
         Lower the order to maintain the correct order in arithmetic
-        operations. If the requestes order is smaller than the current
+        operations. If the requested order is smaller than the current
         order parenthesis will be added to the expression to lower its
         order to 0.
 
@@ -411,7 +411,7 @@ def build(self, arguments: dict) -> BuildAST:
 
         """
         # Game calls are ignored as we have no support for a similar
-        # feature, we simpli return the content inside the GAME call
+        # feature, we simply return the content inside the GAME call
         return arguments["expr"]
 
 

pysd/cli/main.py

@@ -12,6 +12,7 @@
 
 from .parser import parser
 
+
 def main(args):
     """
     Main function. Reads user arguments, loads the models,

pysd/cli/parser.py

@@ -2,6 +2,7 @@
 cmdline parser
 """
 import os
+import re
 from ast import literal_eval
 import numpy as np
 import pandas as pd
@@ -113,15 +114,6 @@ def split_timestamps(string):
             f'See {docs} for examples.')
 
 
-def split_subview_sep(string):
-    """
-    Splits the subview separators
-    --subview-sep ' - ,.' -> [' - ', '.']
-
-    """
-    return string.split(",")
-
-
 def split_vars(string):
     """
     Splits the arguments from new_values.
@@ -141,8 +133,8 @@ def split_vars(string):
             var, value = string.split(':')
             type = 'initial'
 
-        if value.strip().isnumeric():
-            # value is float
+        if re.match(r"^[+-]?(\d*\.)?\d+$", value.strip()):
+            # value is a number
             return {var.strip(): (type, float(value))}
 
         # value is series
@@ -193,7 +185,7 @@ def __call__(self, parser, namespace, values, option_string=None):
 parser.add_argument(
     '-o', '--output-file', dest='output_file',
     type=check_output, metavar='FILE',
-    help='output file to save run outputs (.tab or .csv)')
+    help='output file to save run outputs (.tab, .csv or .nc)')
 
 parser.add_argument(
     '-p', '--progress', dest='progress',
@@ -282,11 +274,12 @@ def __call__(self, parser, namespace, values, option_string=None):
 
 trans_arguments.add_argument(
     '--subview-sep', dest='subview_sep',
-    action='store', type=split_subview_sep, default=[],
-    metavar='\'STRING1,STRING2,..,STRINGN\'',
-    help='further division of views split in subviews, by identifying the'
+    action='store', nargs="*", default=[],
+    metavar='separator_1 separator_2 ... separator_n',
+    help='further division of views into subviews, by identifying the '
          'separator string in the view name, only availabe if --split-views'
-         ' is used')
+         ' is used. Passing positional arguments after this argument will'
+         ' not work')
 
 
 #######################
@@ -316,17 +309,17 @@ def __call__(self, parser, namespace, values, option_string=None):
 parser.add_argument('new_values',
                     metavar='variable=new_value', type=split_vars,
                     nargs='*', action=SplitVarsAction,
-                    help='redefine the value of variable with new value.'
-                    'variable must be a model component, new_value can be a '
-                    'float or a a list of two list')
+                    help='redefine the value of variable with new value. '
+                    'variable must be a model component, new_value may be a '
+                    'float or a list of two lists')
 
-# The destionation new_values2 will never used as the previous argument
+# The destination new_values2 will never be used as the previous argument
 # is given also with nargs='*'. Nevertheless, the following variable
 # is declared for documentation
 parser.add_argument('new_values2',
                     metavar='variable:initial_value', type=split_vars,
                     nargs='*', action=SplitVarsAction,
-                    help='redefine the initial value of variable.'
+                    help='redefine the initial value of variable. '
                     'variable must be a model stateful element, initial_value'
                     ' must be a float')
 

pysd/py_backend/allocation.py

@@ -682,13 +682,11 @@ def allocate_by_priority(request, priority, width, supply):
 
     if np.any(width <= 0):
         raise ValueError(
-            f"width={width} is not allowed. width should be greater than 0."
-        )
+            f"width={width} \n is not allowed. width must be greater than 0.")
 
     if np.any(supply < 0):
         raise ValueError(
-            f"supply={supply} is not allowed. supply should be non-negative."
-        )
+            f"supply={supply} \n is not allowed. supply must not be negative.")
 
     if len(request.shape) == 1:
         # NUMPY: avoid '.values' and return directly the result of the

pysd/translators/structures/abstract_model.py

@@ -312,10 +312,59 @@ def dump(self, depth=None, indent="") -> str:  # pragma: no cover
         return self.__str__()
 
 
+@dataclass
+class AbstractConstraint:
+    """
+    Dataclass for a constraint
+
+    Parameters
+    ----------
+    name: str
+        Name of the constraint variable.
+    subscripts: list or str or dict
+        The subscripts as a list of strings for a regular definition,
+        str for a copy definition and as a dict for a GET XLS/DIRECT
+        definition.
+    expression: str
+        Unparsed constraint expression.
+    """
+
+    name: str
+    subscripts: Union[list, str, dict]
+    expression: str
+
+    def __str__(self) -> str:  # pragma: no cover
+        raise NotImplementedError()
+
+
+@dataclass
+class AbstractTestInput:
+    """
+    Dataclass for a test inputs
+
+    Parameters
+    ----------
+    name: str
+        Name of the test inputs variable.
+    subscripts: list or str or dict
+        The subscripts as a list of strings for a regular definition,
+        str for a copy definition and as a dict for a GET XLS/DIRECT
+        definition.
+    expression: str
+        Unparsed test inputs expression.
+    """
+    name: str
+    subscripts: Union[list, str, dict]
+    expression: str
+
+    def __str__(self) -> str:  # pragma: no cover
+        raise NotImplementedError()
+
+
 @dataclass
 class AbstractSection:
     """
-    Dataclass for an element.
+    Dataclass for a section.
 
     Parameters
     ----------
@@ -336,6 +385,10 @@ class AbstractSection:
         Tuple of AbstractSubscriptRanges that are defined in the section.
     elements: tuple
         Tuple of AbstractElements that are defined in the section.
+    constraints: tuple
+        Tuple of AbstractConstraints that are defined in the section.
+    test_inputs: tuple
+        Tuple of TestInputs that are defined in the section.
     split: bool
         If split is True the created section will split the variables
         depending on the views_dict.
@@ -351,6 +404,8 @@ class AbstractSection:
     returns: List[str]
     subscripts: Tuple[AbstractSubscriptRange]
     elements: Tuple[AbstractElement]
+    constraints: Tuple[AbstractConstraint]
+    test_inputs: Tuple[AbstractTestInput]
     split: bool
     views_dict: Union[dict, None]
 
@@ -390,7 +445,7 @@ def _str_child(self, depth, indent) -> str:  # pragma: no cover
 @dataclass
 class AbstractModel:
     """
-    Dataclass for an element.
+    Dataclass for a model.
 
     Parameters
     ----------

pysd/translators/vensim/parsing_grammars/element_object.peg

@@ -1,6 +1,6 @@
 # Parsing Expression Grammar: element_object
 
-entry = unchangeable_constant / component / data_definition / subscript_definition / lookup_definition / subscript_copy
+entry = constraint_definition / test_inputs_definition / unchangeable_constant / component / data_definition / subscript_definition / lookup_definition / subscript_copy
 
 # Regular component definition "="
 component = name _ subscript_component? _ "=" _ expression
@@ -17,6 +17,12 @@ data_definition = component_data_definition / empty_data_definition
 component_data_definition = name _ subscript_component? _ keyword? _ ":=" _ expression
 empty_data_definition = name _ subscript_component? _ keyword
 
+# Constraint definition
+constraint_definition = name _ subscript_component? _ ":THE CONDITION:" _ expression
+
+# Test inputs definition
+test_inputs_definition = name _ subscript_component? _ ":TEST INPUT:" _ expression
+
 # Subscript ranges
 # Subcript range  regular definition ":"
 subscript_definition = name _ ":" _ (imported_subscript / literal_subscript) _ subscript_mapping_list?