Store all calibrated models in Problem.state; update all notebooks (#…

…134) * fix type error list[Model] -> Models * support provision of subspace petab problem to `Model.from_yaml`. backwards compatibility: add `Model.get_hash`, support `predecessor_model_hash=None` input * backwards compatibility: support `Models` construction from single model YAML * add all iteration calibrated models to problem state in `end_iteration` * add `Problem._state`, currently just stores all calibrated models from all iterations * workflow_python notebook: demo `Models.df`; demo `Problem.state.models` * start tracking iteration in `State` * Update test_cases/0009/README.md --------- Co-authored-by: Daniel Weindl <[email protected]>
PEtab-dev · Jan 6, 2025 · 67c4d79 · 67c4d79
1 parent d4fce46
commit 67c4d79
Show file tree

Hide file tree

Showing 23 changed files with 253 additions and 201 deletions.
diff --git a/doc/examples/README.md b/doc/examples/README.md
@@ -0,0 +1 @@
+These notebooks need to be run to see the output. Pre-computed output can be viewed in the documentation, at https://petab-select.readthedocs.io/en/stable/examples.html
diff --git a/doc/examples/calibrated_models/calibrated_models.yaml b/doc/examples/calibrated_models/calibrated_models.yaml
@@ -15,8 +15,8 @@
     k1: 0
     k2: 0
     k3: 0
-  petab_yaml: petab_problem.yaml
-  predecessor_model_hash: virtual_initial_model
+  model_subspace_petab_yaml: petab_problem.yaml
+  predecessor_model_hash: virtual_initial_model-
 - criteria:
     AICc: -0.17540608110890332
     NLLH: -4.087703040554452
@@ -35,7 +35,7 @@
     k1: 0.2
     k2: 0.1
     k3: estimate
-  petab_yaml: petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   predecessor_model_hash: M_0-000
 - criteria:
     AICc: -0.27451438069575573
@@ -55,7 +55,7 @@
     k1: 0.2
     k2: estimate
     k3: 0
-  petab_yaml: petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   predecessor_model_hash: M_0-000
 - criteria:
     AICc: -0.7053270766271886
@@ -75,7 +75,7 @@
     k1: estimate
     k2: 0.1
     k3: 0
-  petab_yaml: petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   predecessor_model_hash: M_0-000
 - criteria:
     AICc: 9.294672923372811
@@ -96,7 +96,7 @@
     k1: estimate
     k2: 0.1
     k3: estimate
-  petab_yaml: petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   predecessor_model_hash: M_3-000
 - criteria:
     AICc: 7.8521704398854
@@ -117,7 +117,7 @@
     k1: estimate
     k2: estimate
     k3: 0
-  petab_yaml: petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   predecessor_model_hash: M_3-000
 - criteria:
     AICc: 35.94352968170024
@@ -139,5 +139,5 @@
     k1: estimate
     k2: estimate
     k3: estimate
-  petab_yaml: petab_problem.yaml
+  model_subspace_petab_yaml: petab_problem.yaml
   predecessor_model_hash: M_3-000
diff --git a/doc/examples/example_cli_famos.ipynb b/doc/examples/example_cli_famos.ipynb
@@ -38,7 +38,8 @@
     "\n",
     "output_path = Path().resolve() / \"output_famos\"\n",
     "output_path_str = str(output_path)\n",
-    "shutil.rmtree(output_path_str)\n",
+    "if output_path.exists():\n",
+    "    shutil.rmtree(output_path_str)\n",
     "output_path.mkdir(exist_ok=False, parents=True)"
    ]
   },

diff --git a/doc/examples/example_cli_famos_calibration_tool.py b/doc/examples/example_cli_famos_calibration_tool.py
@@ -12,7 +12,7 @@
 for model in models:
     calibrate(model=model)
     predecessor_model_hashes |= {model.predecessor_model_hash}
-models.to_yaml(output_yaml=calibrated_models_yaml)
+models.to_yaml(filename=calibrated_models_yaml)
 
 if len(predecessor_model_hashes) == 0:
     pass

diff --git a/doc/examples/model_selection/calibrated_M1_4.yaml b/doc/examples/model_selection/calibrated_M1_4.yaml
@@ -1,19 +1,19 @@
+model_subspace_id: M1_4
+model_subspace_indices:
+- 0
+- 0
+- 0
 criteria:
   AIC: 15
+model_hash: M1_4-000
+model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
 estimated_parameters:
   k2: 0.15
   k3: 0.0
 iteration: 1
-model_hash: M1_4-000
 model_id: M1_4-000
-model_subspace_id: M1_4
-model_subspace_indices:
-- 0
-- 0
-- 0
 parameters:
   k1: 0
   k2: estimate
   k3: estimate
-petab_yaml: ../model_selection/petab_problem.yaml
-predecessor_model_hash: null
+predecessor_model_hash: M1_2-000
diff --git a/doc/examples/model_selection/calibrated_M1_7.yaml b/doc/examples/model_selection/calibrated_M1_7.yaml
@@ -1,20 +1,20 @@
+model_subspace_id: M1_7
+model_subspace_indices:
+- 0
+- 0
+- 0
 criteria:
   AIC: 20
+model_hash: M1_7-000
+model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
 estimated_parameters:
   k1: 0.25
   k2: 0.1
   k3: 0.0
 iteration: 2
-model_hash: M1_7-000
 model_id: M1_7-000
-model_subspace_id: M1_7
-model_subspace_indices:
-- 0
-- 0
-- 0
 parameters:
   k1: estimate
   k2: estimate
   k3: estimate
-petab_yaml: ../model_selection/petab_problem.yaml
-predecessor_model_hash: null
+predecessor_model_hash: M1_4-000
diff --git a/doc/examples/model_selection/calibrated_models_1.yaml b/doc/examples/model_selection/calibrated_models_1.yaml
@@ -1,51 +1,52 @@
-- criteria:
-    AIC: 180
-  estimated_parameters: {}
-  iteration: 1
-  model_hash: M1_0-000
-  model_id: M1_0-000
-  model_subspace_id: M1_0
+- model_subspace_id: M1_0
   model_subspace_indices:
   - 0
   - 0
   - 0
+  criteria:
+    AIC: 180
+  model_hash: M1_0-000
+  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
+  estimated_parameters: {}
+  iteration: 1
+  model_id: M1_0-000
   parameters:
     k1: 0
     k2: 0
     k3: 0
-  petab_yaml: ../model_selection/petab_problem.yaml
-  predecessor_model_hash: null
-- criteria:
-    AIC: 100
-  estimated_parameters: {}
-  iteration: 1
-  model_hash: M1_1-000
-  model_id: M1_1-000
-  model_subspace_id: M1_1
+  predecessor_model_hash: virtual_initial_model-
+- model_subspace_id: M1_1
   model_subspace_indices:
   - 0
   - 0
   - 0
+  criteria:
+    AIC: 100
+  model_hash: M1_1-000
+  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
+  estimated_parameters: {}
+  iteration: 1
+  model_id: M1_1-000
   parameters:
     k1: 0.2
     k2: 0.1
     k3: estimate
-  petab_yaml: ../model_selection/petab_problem.yaml
-  predecessor_model_hash: null
-- criteria:
-    AIC: 50
-  estimated_parameters: {}
-  iteration: 1
-  model_hash: M1_2-000
-  model_id: M1_2-000
-  model_subspace_id: M1_2
+  predecessor_model_hash: virtual_initial_model-
+
+- model_subspace_id: M1_2
   model_subspace_indices:
   - 0
   - 0
   - 0
+  criteria:
+    AIC: 50
+  model_hash: M1_2-000
+  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
+  estimated_parameters: {}
+  iteration: 1
+  model_id: M1_2-000
   parameters:
     k1: 0.2
     k2: estimate
     k3: 0
-  petab_yaml: ../model_selection/petab_problem.yaml
-  predecessor_model_hash: null
+  predecessor_model_hash: virtual_initial_model-
diff --git a/doc/examples/visualization.ipynb b/doc/examples/visualization.ipynb
@@ -31,7 +31,7 @@
     "\n",
     "import petab_select\n",
     "import petab_select.plot\n",
-    "from petab_select import VIRTUAL_INITIAL_MODEL_HASH\n",
+    "from petab_select import VIRTUAL_INITIAL_MODEL\n",
     "\n",
     "models = petab_select.Models.from_yaml(\n",
     "    \"calibrated_models/calibrated_models.yaml\"\n",
@@ -69,12 +69,12 @@
     "# Custom labels\n",
     "labels = {}\n",
     "for model in models:\n",
-    "    labels[model.get_hash()] = \"M_\" + \"\".join(\n",
+    "    labels[model.hash] = \"M_\" + \"\".join(\n",
     "        \"1\" if value == petab_select.ESTIMATE else \"0\"\n",
     "        for value in model.parameters.values()\n",
     "    )\n",
-    "labels[VIRTUAL_INITIAL_MODEL_HASH] = \"\\n\".join(\n",
-    "    petab_select.VIRTUAL_INITIAL_MODEL.split(\"_\")\n",
+    "labels[VIRTUAL_INITIAL_MODEL.hash] = \"\\n\".join(\n",
+    "    petab_select.VIRTUAL_INITIAL_MODEL.model_subspace_id.split(\"_\")\n",
     ").title()\n",
     "\n",
     "# Custom colors for some models\n",

diff --git a/doc/examples/workflow_python.ipynb b/doc/examples/workflow_python.ipynb
@@ -65,12 +65,12 @@
     "    print(\n",
     "        f\"\"\"\\\n",
     "Model subspace ID: {model.model_subspace_id}\n",
-    "PEtab YAML location: {model.petab_yaml}\n",
+    "PEtab YAML location: {model.model_subspace_petab_yaml}\n",
     "Custom model parameters: {model.parameters}\n",
-    "Model hash: {model.get_hash()}\n",
+    "Model hash: {model.hash}\n",
     "Model ID: {model.model_id}\n",
     "{select_problem.criterion}: {model.get_criterion(select_problem.criterion, compute=False)}\n",
-    "Model calibrated in iteration: {model.iteration}\n",
+    "Model was calibrated in iteration: {model.iteration}\n",
     "\"\"\"\n",
     "    )\n",
     "\n",
@@ -177,6 +177,7 @@
     "    calibrate(candidate_model)\n",
     "\n",
     "iteration_results = petab_select.ui.end_iteration(\n",
+    "    problem=select_problem,\n",
     "    candidate_space=iteration[CANDIDATE_SPACE],\n",
     "    calibrated_models=iteration[UNCALIBRATED_MODELS],\n",
     ")"
@@ -235,6 +236,7 @@
     "\n",
     "    # Finalize iteration\n",
     "    iteration_results = petab_select.ui.end_iteration(\n",
+    "        problem=select_problem,\n",
     "        candidate_space=iteration[CANDIDATE_SPACE],\n",
     "        calibrated_models=iteration[UNCALIBRATED_MODELS],\n",
     "    )\n",
@@ -257,7 +259,7 @@
     ")\n",
     "\n",
     "for candidate_model in iteration_results[MODELS]:\n",
-    "    if candidate_model.get_hash() == local_best_model.get_hash():\n",
+    "    if candidate_model.hash == local_best_model.hash:\n",
     "        print(BOLD_TEXT + \"BEST MODEL OF CURRENT ITERATION\" + NORMAL_TEXT)\n",
     "    print_model(candidate_model)"
    ]
@@ -285,7 +287,7 @@
     ")\n",
     "\n",
     "for candidate_model in iteration_results[MODELS]:\n",
-    "    if candidate_model.get_hash() == local_best_model.get_hash():\n",
+    "    if candidate_model.hash == local_best_model.hash:\n",
     "        print(BOLD_TEXT + \"BEST MODEL OF CURRENT ITERATION\" + NORMAL_TEXT)\n",
     "    print_model(candidate_model)"
    ]
@@ -313,7 +315,7 @@
     ")\n",
     "\n",
     "for candidate_model in iteration_results[MODELS]:\n",
-    "    if candidate_model.get_hash() == local_best_model.get_hash():\n",
+    "    if candidate_model.hash == local_best_model.hash:\n",
     "        print(BOLD_TEXT + \"BEST MODEL OF CURRENT ITERATION\" + NORMAL_TEXT)\n",
     "    print_model(candidate_model)"
    ]
@@ -398,7 +400,7 @@
     "    criterion=select_problem.criterion\n",
     ")\n",
     "# candidate_space.calibrated_models = iteration_results[CANDIDATE_SPACE].calibrated_models\n",
-    "petab_select.ui.start_iteration(\n",
+    "iteration = petab_select.ui.start_iteration(\n",
     "    problem=select_problem,\n",
     "    candidate_space=candidate_space,\n",
     ");"
@@ -411,9 +413,44 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "for candidate_model in candidate_space.models:\n",
+    "for candidate_model in iteration[UNCALIBRATED_MODELS]:\n",
+    "    calibrate(candidate_model)\n",
     "    print_model(candidate_model)"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "924ee6af-f72a-455d-8985-13a6f00a14a1",
+   "metadata": {},
+   "source": [
+    "All calibrated models across all iterations can be retrieved via `Problem.state.models`, which is set by `ui.end_iteration`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10b62dd7-a3c3-420e-a88d-9ac44e367145",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Add latest model to all models\n",
+    "petab_select.ui.end_iteration(\n",
+    "    problem=select_problem,\n",
+    "    candidate_space=iteration[CANDIDATE_SPACE],\n",
+    "    calibrated_models=iteration[UNCALIBRATED_MODELS],\n",
+    ");"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "68d1f89a-31a2-4f96-8fe0-82d1ad130832",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Print all models\n",
+    "select_problem.state.models.df"
+   ]
   }
  ],
  "metadata": {

diff --git a/doc/standard/model.yaml b/doc/standard/model.yaml
@@ -1,10 +1,7 @@
 $defs:
   ModelHash:
     type: string
-description: "A model.\n\nSee :class:`ModelBase` for the standardized attributes.\
-  \ Additional\nattributes are available in ``Model`` to improve usability.\n\nAttributes:\n\
-  \    _model_subspace_petab_problem:\n        The PEtab problem of the model subspace\
-  \ of this model.\n        If not provided, this is reconstructed from\n        :attr:`model_subspace_petab_yaml`."
+description: A model.
 properties:
   model_subspace_id:
     title: Model Subspace Id
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		These notebooks need to be run to see the output. Pre-computed output can be viewed in the documentation, at https://petab-select.readthedocs.io/en/stable/examples.html