Merge pull request #325 from JamesPHoughton/dev

General improvements

.github/workflows/ci.yml

@@ -10,7 +10,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest, windows-latest]
-        python-version: [3.7, 3.9]
+        python-version: ['3.7', '3.10']
 
     steps:
       - uses: actions/checkout@v2

docs/conf.py

@@ -151,7 +151,8 @@
     'pysdcookbook': ('http://pysd-cookbook.readthedocs.org/en/latest/', None),
     'pandas': ('https://pandas.pydata.org/docs/', None),
     'xarray': ('https://docs.xarray.dev/en/stable/', None),
-    'numpy': ('https://numpy.org/doc/stable/', None)
+    'numpy': ('https://numpy.org/doc/stable/', None),
+    'pytest': ('https://docs.pytest.org/en/7.1.x/', None)
 }
 
 # -- Options for autodoc --------------------------------------------------

docs/development/guidelines.rst

@@ -19,24 +19,24 @@ There are a number of tools that you might find helpful in development:
 Test Suite
 ^^^^^^^^^^
 PySD uses the common model test suite found `on github <https://github.com/SDXorg/test-models>`_
-which are run using `integration_test_vensim_pathway.py` and `integration_test_xmile_pathway.py`.
-PySD also has own tests for internal funtionality, `unit_test_*.py` files
+which are run using `pytest_integration_test_vensim_pathway.py`  and `pytest_integration_test_xmile_pathway.py`.
+PySD also has own tests for internal funtionality, `pytest_*.py` files
 of the `/tests/` directory.
 
-In order to run all the tests :py:mod:`nose` or :py:mod:`pytest` can be used.
+In order to run all the tests :py:mod:`pytest` should be used.
 A `Makefile` is given to run easier the tests with :py:mod:`pytest`, check
 `tests/README <https://github.com/JamesPHoughton/pysd/tree/master/tests/README.md>`_
 for more information.
 
 These tests run quickly and should be executed when any changes are made to ensure
 that current functionality remains intact. If any new functionality is added or a
 bug is corrected, the tests should be updated with new models in test suite or
-complementary tests in the corresponding `unit_test_*.py` file.
+complementary tests in the corresponding `pytest_*.py` file.
 
 .. note::
   If your changes correct some existing bug related to the translation or running
-  of a Vensim (or Xmile) model. You should add a new test in the `test suite repo <https://github.com/SDXorg/test-models>`_ reproducing the solved bug and addthe necessary lines in `integration_test_vensim_pathway.py` (or `integration_test_xmile_pathway.py`) to run the new test. Then, it is encoraged to add also unit test in `unit_test_vensim2py.py` (or `unit_test_xmile2py.py`) reproducing the translation of the new function and test of the workflow in
-  `unit_test_functions.py` (or `unit_test_statefuls.py`).
+  of a Vensim (or Xmile) model. You should add a new test in the `test suite repo <https://github.com/SDXorg/test-models>`_ reproducing the solved bug and addthe necessary lines in `pytest_integration/pytest_integration_test_vensim_pathway.py` (or `pytest_integration/pytest_integration_test_xmile_pathway.py`) to run the new test. Then, it is encoraged to add also unit test in `pytest_translators` reproducing the translation of the new function and test of the workflow in
+  `pytest_types/functions/pytest_functions.py` (or `pytest_types/statefuls/pytest_statefuls.py`).
 
 Speed Tests
 ^^^^^^^^^^^

docs/development/pathway.rst

@@ -9,7 +9,6 @@ High Priority
 -------------
 * Improve running speed using numpy.arrays instead of xarray.DataArrays
 * Adding unit and full tests for Xmile translation
-* Subscripts/arrays support for Xmile models
 
 
 Medium Priority

docs/structure/python_builder.rst

@@ -7,6 +7,8 @@ The use of a one-to-one dictionary in translation means that the breadth of func
 
 In addition to translating individual commands between Vensim/XMILE and Python, PySD reworks component identifiers to be Python-safe by replacing spaces with underscores. The translator allows source identifiers to make use of alphanumeric characters, spaces, or special characteres. In order to make that possible a namespace is created, which links the original name of the variable with the Python-safe name. The namespace is also available in the PySD model class to allow users working with both original names and Python-safe names.
 
+The resulting Python code from building the model is formatted with :py:mod:`black` library and it is written to the same folder where the original model is.
+
 
 Main builders
 -------------

docs/whats_new.rst

@@ -1,8 +1,42 @@
 
 What's New
 ==========
+v3.0.1 (2022/05/26)
+-------------------
+
+New Features
+~~~~~~~~~~~~
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+Deprecations
+~~~~~~~~~~~~
+
+Bug fixes
+~~~~~~~~~
+
+- Simplify subscripts dictionaries for :py:class:`pysd.py_backend.data.TabData` objects.
+
+Documentation
+~~~~~~~~~~~~~
+- Improve tests/README.md.
+- Minor improvements in the documentation.
+
+Performance
+~~~~~~~~~~~
+
+Internal Changes
+~~~~~~~~~~~~~~~~
+- Add Python 3.10 to CI pipeline and include it in the supported versions list.
+- Correct LICENSE file extension in the `setup.py`.
+- Move from `importlib`'s :py:func:`load_module` to :py:func:`exec_module`.
+- Remove warnings related to :py:data:`set` usage.
+- Move all the missing test to :py:mod:`pytest`.
+- Remove warning messages from test and make test fail if there is any warning.
+
 
-v3.0.0 (23/05/2022)
+v3.0.0 (2022/05/23)
 -------------------
 
 New Features

pysd/_version.py

@@ -1 +1 @@
-__version__ = "3.0.0"
+__version__ = "3.0.1"

pysd/builders/python/python_expressions_builder.py

@@ -990,7 +990,6 @@ def build(self, arguments: dict) -> BuildAST:
         arguments["real_name"] = self.element.name
         arguments["py_name"] =\
             self.section.namespace.namespace[self.element.name]
-        arguments["subscripts"] = self.def_subs
         arguments["method"] = "'%s'" % self.keyword if self.keyword else None
 
         arguments["name"] = self.section.namespace.make_python_identifier(

pysd/cli/parser.py

@@ -109,7 +109,7 @@ def split_timestamps(string):
         # error
         raise parser.error(
             f'when parsing {string}'
-            '\nThe return time stamps much be separated by commas...\n'
+            '\nThe return time stamps must be separated by commas...\n'
             f'See {docs} for examples.')
 
 

pysd/py_backend/components.py

@@ -6,7 +6,7 @@
 import os
 import random
 import inspect
-from importlib.machinery import SourceFileLoader
+import importlib.util
 
 import numpy as np
 
@@ -73,8 +73,11 @@ def _load(self, py_model_file):
         module_name = os.path.splitext(py_model_file)[0]\
             + str(random.randint(0, 1000000))
         try:
-            return SourceFileLoader(
-                module_name, py_model_file).load_module()
+            spec = importlib.util.spec_from_file_location(
+                module_name, py_model_file)
+            module = importlib.util.module_from_spec(spec)
+            spec.loader.exec_module(module)
+            return module
         except TypeError:
             raise ImportError(
                 "\n\nNot able to import the model. "

pysd/py_backend/data.py

@@ -46,9 +46,9 @@ def read_file(cls, file_name, encoding=None):
 
         Returns
         -------
-        out, transposed: set, bool
-            The set of the columns in the output file and a boolean flag to
-            indicate if the output file is transposed.
+        out, transposed: list, bool
+            The list of the columns in the output file and a boolean flag
+            to indicate if the output file is transposed.
 
         """
         # in the most cases variables will be split per columns, then
@@ -85,35 +85,35 @@ def read_line(cls, file_name, encoding=None):
         Read the firts row and return a set of it.
         """
         if file_name.suffix.lower() == ".tab":
-            return set(pd.read_table(file_name,
-                                     nrows=0,
-                                     encoding=encoding,
-                                     dtype=str,
-                                     header=0).iloc[:, 1:])
+            return list(pd.read_table(file_name,
+                                      nrows=0,
+                                      encoding=encoding,
+                                      dtype=str,
+                                      header=0).iloc[:, 1:])
         elif file_name.suffix.lower() == ".csv":
-            return set(pd.read_csv(file_name,
-                                   nrows=0,
-                                   encoding=encoding,
-                                   dtype=str,
-                                   header=0).iloc[:, 1:])
+            return list(pd.read_csv(file_name,
+                                    nrows=0,
+                                    encoding=encoding,
+                                    dtype=str,
+                                    header=0).iloc[:, 1:])
         else:
             return None
 
     @classmethod
     def read_col(cls, file_name, encoding=None):
         """
-        Read the firts column and return a set of it.
+        Read the firts column and return a it.
         """
         if file_name.suffix.lower() == ".tab":
-            return set(pd.read_table(file_name,
-                                     usecols=[0],
-                                     encoding=encoding,
-                                     dtype=str).iloc[:, 0].to_list())
+            return list(pd.read_table(file_name,
+                                      usecols=[0],
+                                      encoding=encoding,
+                                      dtype=str).iloc[:, 0].to_list())
         elif file_name.suffix.lower() == ".csv":
-            return set(pd.read_csv(file_name,
-                                   usecols=[0],
-                                   encoding=encoding,
-                                   dtype=str).iloc[:, 0].to_list())
+            return list(pd.read_csv(file_name,
+                                    usecols=[0],
+                                    encoding=encoding,
+                                    dtype=str).iloc[:, 0].to_list())
 
     @classmethod
     def get_columns(cls, file_name, vars=None, encoding=None):

pysd/py_backend/external.py

@@ -710,7 +710,7 @@ def __init__(self, file_name, sheet, time_row_or_col, cell,
             raise ValueError(self.py_name + "\n"
                              + " The interpolation method (interp) must be "
                              + "'raw', 'interpolate', "
-                             + "'look_forward' or 'hold_backward")
+                             + "'look_forward' or 'hold_backward'")
 
     def add(self, file_name, sheet, time_row_or_col, cell,
             interp, coords):

pysd/py_backend/utils.py

@@ -428,7 +428,7 @@ def load_outputs(file_name, transpose=False, columns=None, encoding=None):
                            encoding=encoding,
                            index_col=0).T
                 if columns:
-                    out = out[columns]
+                    out = out[list(columns)]
             else:
                 out = func(file_name,
                            encoding=encoding,
@@ -460,8 +460,9 @@ def detect_encoding(filename):
 
     """
     detector = UniversalDetector()
-    for line in open(filename, 'rb').readlines():
-        detector.feed(line)
+    with open(filename, 'rb') as file:
+        for line in file.readlines():
+            detector.feed(line)
     detector.close()
     return detector.result['encoding']
 
@@ -483,16 +484,16 @@ class ProgressBar:
     Progress bar for integration
     """
 
-    def __init__(self, maxval=None):
+    def __init__(self, max_value=None):
 
-        self.maxval = maxval
-        if self.maxval is None:
+        self.max_value = max_value
+        if self.max_value is None:
             return
 
         self.counter = 0
 
         self.bar = progressbar.ProgressBar(
-            maxval=self.maxval,
+            max_value=self.max_value,
             widgets=[
                 progressbar.ETA(),
                 " ",

pysd/tools/benchmarking.py

@@ -296,4 +296,5 @@ def _remove_constant_nan(df):
     Removes nana values in constant value columns produced by Vensim
     """
     nan_cols = np.isnan(df.iloc[1:, :]).all()
-    df.loc[:, nan_cols] = df.loc[:, nan_cols].iloc[0].values
+    cols = nan_cols[nan_cols].index
+    df[cols] = df[cols].apply(lambda x: x.iloc[0])

setup.py

@@ -11,13 +11,13 @@
     author_email='[email protected]',
     packages=find_packages(exclude=['docs', 'tests', 'dist', 'build']),
     url='https://github.com/JamesPHoughton/pysd',
-    license='LICENSE.txt',
+    license='LICENSE',
     description='System Dynamics Modeling in Python',
     long_description=open('README.md').read(),
     long_description_content_type='text/markdown',
     keywords=['System Dynamics', 'Vensim', 'XMILE'],
     classifiers=[
-        'Development Status :: 3 - Alpha',
+        'Development Status :: 4 - Beta',
         'License :: OSI Approved :: MIT License',
         'Topic :: Scientific/Engineering :: Mathematics',
         'Topic :: Scientific/Engineering :: Information Analysis',
@@ -26,6 +26,7 @@
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
         'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
     ],
     install_requires=open('requirements.txt').read().strip().split('\n'),
     package_data={

tests/README.md

@@ -1,23 +1,48 @@
-Tests use [unittest](https://docs.python.org/3/library/unittest.html), [pytest](https://docs.pytest.org/en/stable/), [pytest-cov](https://pytest-cov.readthedocs.io/en/latest/) and [pytest-xdist](https://docs.pytest.org/en/2.1.0/xdist.html).
+Test suite
+==========
+
+Preparing the test suite
+------------------------
+
+In order to run the test:
+
+1. Clone the repo with the --recursive flag (if not done yet).
+2. Potentially create or enter a virtual environment.
+3. Install test dependencies using *pip install -r tests/requirements.txt* or *conda install --file tests/requirements.txt*.
+
+Running the test suite
+----------------------
 
 To run tests:
-```
+
+```shell
 make tests
 ```
 
 To have coverage information:
-```
+
+```shell
 make cover
 ```
 
 To have information of visit lines in HTML files in `htmlcov` folder:
-```
+
+```shell
 make coverhtml
 ```
 
 To run in multiple CPUs (e.g: for 4 CPUs):
-```
+
+```shell
 make command NUM_PROC=4
 ```
+
 where `command` is any of above.
 
+You can also run the test using `pytest`:
+
+```shell
+python -m pytest test/
+```
+
+for running individual test, filtering warnings or some other configurations check [pytest documentation](https://docs.pytest.org).

tests/pytest.ini

@@ -1,2 +1,5 @@
 [pytest]
-python_files = unit_test_*.py pytest_*/**/*.py pytest_*/*.py
+python_files = pytest_*/**/*.py pytest_*/*.py
+filterwarnings =
+    error
+    ignore:Creating an ndarray from ragged nested sequences