Release 0.1.0 (#26)

Co-authored-by: Dilan Pathirana <[email protected]>
Co-authored-by: Yannik Schälte <[email protected]>
Co-authored-by: Doresic <[email protected]>

.flake8

@@ -0,0 +1,9 @@
+########################
+# Flake8 Configuration #
+########################
+
+[flake8]
+
+per-file-ignores =
+    # Imported but unused
+    */__init__.py:F401,F403

.github/workflows/ci.yml

@@ -0,0 +1,56 @@
+name: CI
+
+# trigger
+on:
+  push:
+    branches:
+    - main
+  pull_request:
+  schedule:
+    # run Tuesday and Friday at 02:00 UTC
+    - cron: '00 2 * * TUE,FRI'
+
+jobs:
+  base:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        # test on latest and minimum python version
+        python-version: ['3.10', '3.8']
+
+    steps:
+    - name: Check out repository
+      uses: actions/checkout@v2
+
+    - name: Prepare python ${{ matrix.python-version }}
+      uses: actions/setup-python@v1
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Cache
+      uses: actions/cache@v1
+      with:
+        path: ~ /.cache
+        key: ci-${{ runner.os }}-${{ matrix.python-version }}-base
+
+    - name: Install system dependencies
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y \
+          swig \
+          libatlas-base-dev \
+          libhdf5-serial-dev
+
+    - name: Install Python dependencies
+      run: pip install -r requirements_dev.txt
+
+    - name: Run tox
+      run: python -m tox
+
+    - name: Run pre-commit hooks
+      run: pre-commit run --all-files
+
+    - name: Coverage
+      uses: codecov/codecov-action@v1
+      with:
+        file: ./coverage.xml

.gitignore

@@ -2,3 +2,14 @@ __pycache__
 .ipynb_checkpoints
 dist
 amici_models
+*egg-info
+
+# coverage
+.coverage*
+coverage.xml
+
+# tox
+.tox
+
+# osx
+.DS_Store

.pre-commit-config.yaml

@@ -0,0 +1,46 @@
+# This is run as a precondition to commits, run manually via `pre-commit run`
+
+# When adding new hooks, it may make sense to once run
+# `pre-commit run --all-files` as by default only changed files are checked
+
+repos:
+- repo: https://github.com/psf/black
+  rev: 22.6.0
+  hooks:
+  - id: black
+    description: The uncompromising code formatter
+- repo: https://github.com/pycqa/isort
+  rev: 5.10.1
+  hooks:
+  - id: isort
+    name: isort (python)
+  - id: isort
+    name: isort (cython)
+    types: [cython]
+  - id: isort
+    name: isort (pyi)
+    types: [pyi]
+- repo: https://github.com/nbQA-dev/nbQA
+  rev: 1.3.1
+  hooks:
+    - id: nbqa-black
+    - id: nbqa-pyupgrade
+      args: [--py36-plus]
+    - id: nbqa-isort
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v4.3.0
+  hooks:
+  - id: check-yaml
+    description: Check yaml files for parseable syntax
+  - id: check-added-large-files
+    description: Prevent large files from being committed
+  - id: check-merge-conflict
+    description: Check for files that contain merge conflict strings
+  - id: check-symlinks
+    description: Check for symlinks which do not point to anything
+  - id: trailing-whitespace
+    description: Trim trailing whitespaces
+  - id: end-of-file-fixer
+    description: Fix empty lines at ends of files
+  - id: detect-private-key
+    description: Detects the presence of private keys

changes.md

@@ -1,5 +1,60 @@
 # Changes
 
+## 0.1.0
+There are several new candidate spaces, including the FAMoS method. There are also code quality checks (`black` / `isort` / notebook checks) and tests / CI with `tox`.
+
+Below are some significant changes. One major change is that the predecessor model is no longer explicitly specified, instead it is taken as the best model from a set of provided models.
+This means a calibration tool only needs to calibrate all models that PEtab Select provides, then send all calibrated models back to PEtab select, at each iteration.
+This change is implemented to support the FAMoS method.
+
+### `CandidateSpace`
+- new candidate spaces
+  - `BidirectionalCandidateSpace`
+    - tests models with both the forward and backward method at each iteration
+  - `ForwardAndBackwardCandidateSpace`
+    - alternates between the forward and backward method
+  - `LateralCandidateSpace`
+    - moves "sideways" through the model space with parameter swaps
+      - a lateral move is where one parameter is turned off, and another parameter is turned on, simultaneously
+  - `FamosCandidateSpace`
+    - implements the FAMoS method ( https://doi.org/10.1371/journal.pcbi.1007230 )
+- candidate spaces have a `update_after_calibration` method
+  - this provides functionality to support FAMoS operations that occur inbetween calibration iterations
+
+### `Problem`
+- renamed method `add_calibrated_models` to `exclude_models`
+- new method `exclude_model_hashes`
+- cleaned `get_best`
+
+### `ui.candidates`
+- logic change: predecessor model is taken from the following in descending priority, to support the FAMoS method
+  - best of `newly_calibarated_models`
+  - best of `calibrated_models`
+  - `previous_predecessor_model`
+- refactored to instead take newly calibrated models as an argument, instead of assuming they are at `candidate_space.models`
+- renamed `history` to `calibrated_models`
+
+### CLI
+- `candidates`
+  - see logic change in `ui.candidates`
+    - breaking change: renamed `--predecessor` to `--previous-predecessor-model` to ensure this logic change is noticed by developers
+    - removed deprecated `--initial` argument
+    - removed `--best`
+    - added argument `--calibrated-models` to specify file(s) that contain all calibrated models from all iterations so far
+    - added argument `--newly-calibrated-models` to specify file(s) that contain all calibrates models from the latest iteration
+  - renamed arguments to align `cli.candidates` with `ui.candidates`
+    - renamed `--yaml` to `--problem`
+    - renamed `--excluded-model-file` to `--excluded-model-file`
+    - renamed `--excluded-model-hash-file` to `--excluded-model-hashes`
+- `best`
+  - renamed arguments to align `cli.best` with `ui.best`
+    - renamed `--yaml` to `--problem`
+    - renamed `--models_yaml` to `--models`
+- `model_to_petab` and `models_to_petab`, respectively
+  - renamed to align `ui` and `cli` methods
+    - renamed `--yaml` to `model` and `models`, respectively
+    - renamed shorthand `-m` for `--model_id` to `-i` to avoid conflict with new `--model/--models` arguments
+
 ## 0.0.9
 Fixed automatic exclusions from the Python interface. Now, exclusions are only automatically managed in `Problem`s (thereby also the model space and model subspaces) and `CandidateSpace`s.
 

doc/examples/example_cli_famos.ipynb

@@ -0,0 +1,209 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "e28947de",
+   "metadata": {},
+   "source": [
+    "# FAMoS in PEtab Select\n",
+    "\n",
+    "In this notebook, the FAMoS algorithm [1] is demonstrated.\n",
+    "\n",
+    "[1] Gabel M, Hohl T, Imle A, Fackler OT, Graw F (2019) FAMoS: A Flexible and dynamic Algorithm for Model Selection to analyse complex systems dynamics. PLOS Computational Biology 15(8): e1007230. https://doi.org/10.1371/journal.pcbi.1007230"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "484b7f3c",
+   "metadata": {},
+   "source": [
+    "The model space contains 65536 models. In normal PEtab Select workflows, a calibration tool would take the place of the `example_cli_famos_calibration_tool.py` script. This script emulates a calibration tool: it takes PEtab Select models and assigns criterion values to them, based on previous calibration results for the same models."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "1f04dce0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cleanup the state and candidate models output by a previous run of this notebook\n",
+    "import shutil\n",
+    "from pathlib import Path\n",
+    "\n",
+    "from example_cli_famos_helpers import (\n",
+    "    expected_criterion_values,\n",
+    "    parse_summary_to_progress_list,\n",
+    "    petab_select_problem_yaml,\n",
+    ")\n",
+    "\n",
+    "output_path = Path().resolve() / \"output_famos\"\n",
+    "output_path_str = str(output_path)\n",
+    "shutil.rmtree(output_path_str)\n",
+    "output_path.mkdir(exist_ok=False, parents=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "a81560e6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import petab\n",
+    "\n",
+    "import petab_select\n",
+    "from petab_select import ESTIMATE, FamosCandidateSpace, Method, Model\n",
+    "from petab_select.constants import Criterion\n",
+    "from petab_select.model import default_compare\n",
+    "\n",
+    "state = str(output_path / 'state.dill')\n",
+    "\n",
+    "# Each iteration of model selection is described as a 2-tuple here.\n",
+    "# First value is the model selection method.\n",
+    "# Second value are relative change in parameter indices that correspond\n",
+    "# to the best model from this iteration.\n",
+    "# e.g. `(Method.FORWARD, {3})` states that the best model from a forward move\n",
+    "# is the model that now estimates the parameter at index 3.\n",
+    "expected_progress_list = [\n",
+    "    (Method.LATERAL, set()),\n",
+    "    (Method.LATERAL, {4, 15}),\n",
+    "    (Method.LATERAL, {9, 13}),\n",
+    "    (Method.FORWARD, set()),\n",
+    "    (Method.FORWARD, {3}),\n",
+    "    (Method.FORWARD, {11}),\n",
+    "    (Method.BACKWARD, set()),\n",
+    "    (Method.BACKWARD, {6}),\n",
+    "    (Method.BACKWARD, {10}),\n",
+    "    (Method.BACKWARD, {8}),\n",
+    "    (Method.BACKWARD, {14}),\n",
+    "    (Method.BACKWARD, {1}),\n",
+    "    (Method.BACKWARD, {16}),\n",
+    "    (Method.BACKWARD, {4}),\n",
+    "    (Method.FORWARD, set()),\n",
+    "    (Method.LATERAL, set()),\n",
+    "    (Method.MOST_DISTANT, {2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15}),\n",
+    "    (Method.LATERAL, {16, 7}),\n",
+    "    (Method.LATERAL, {5, 12}),\n",
+    "    (Method.LATERAL, {13, 15}),\n",
+    "    (Method.LATERAL, {1, 6}),\n",
+    "    (Method.FORWARD, set()),\n",
+    "    (Method.FORWARD, {3}),\n",
+    "    (Method.FORWARD, {7}),\n",
+    "    (Method.FORWARD, {2}),\n",
+    "    (Method.FORWARD, {11}),\n",
+    "    (Method.BACKWARD, set()),\n",
+    "    (Method.BACKWARD, {7}),\n",
+    "    (Method.BACKWARD, {16}),\n",
+    "    (Method.BACKWARD, {4}),\n",
+    "    (Method.FORWARD, set()),\n",
+    "    (Method.LATERAL, set()),\n",
+    "    (Method.LATERAL, {9, 15}),\n",
+    "    (Method.FORWARD, set()),\n",
+    "    (Method.BACKWARD, set()),\n",
+    "    (Method.LATERAL, set()),\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7202f6c6",
+   "metadata": {},
+   "source": [
+    "The predecessor model is some model from the model space, and is defined in the PEtab Select problem YAML file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "bb1a5144",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash -s \"$petab_select_problem_yaml\" \"$output_path_str\"\n",
+    "\n",
+    "petab_select_problem_yaml=$1\n",
+    "output_path_str=$2\n",
+    "\n",
+    "problem=$petab_select_problem_yaml\n",
+    "state=$output_path_str/state.dill\n",
+    "\n",
+    "newly_calibrated_models=\"\"\n",
+    "calibrated_models=\"\"\n",
+    "empty_output=$(echo -e \"null\\n...\")\n",
+    "\n",
+    "for i in {1..40}\n",
+    "do\n",
+    "\n",
+    "output=$output_path_str/models_$i.yaml\n",
+    "calibrated_output=$output_path_str/calibrated_models_$i.yaml\n",
+    "\n",
+    "petab_select candidates \\\n",
+    "--problem $problem \\\n",
+    "--state $state \\\n",
+    "--output $output \\\n",
+    "$newly_calibrated_models \\\n",
+    "$calibrated_models \\\n",
+    "--relative-paths\n",
+    "\n",
+    "# Replace this line with a tool that calibrates the models.\n",
+    "# The script also changes model IDs for easier analysis in this example.\n",
+    "python example_cli_famos_calibration_tool.py $output $calibrated_output\n",
+    "\n",
+    "if [ \"$(cat $calibrated_output)\" == \"$empty_output\" ]\n",
+    "then\n",
+    "        # End the model selection if no models were provided by PEtab Select.\n",
+    "        break\n",
+    "else\n",
+    "        newly_calibrated_models=\"--newly-calibrated-models \"$calibrated_output\n",
+    "        calibrated_models+=\"--calibrated-models \"$calibrated_output\" \"\n",
+    "fi\n",
+    "\n",
+    "done"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "93caf071",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "progress_list = parse_summary_to_progress_list('output_famos/summary.tsv')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "cb61d0f7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assert progress_list == expected_progress_list"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}