extend documentation

Pipfile

@@ -30,6 +30,7 @@ sphinx-sitemap = "*"
 sphinx-design = "*"
 sphinx-copybutton = "*"
 sphinxcontrib-typer = "*"
+autodoc_pydantic = "*"
 #sphinx-autodoc2  # TODO: https://myst-parser.readthedocs.io/en/latest/intro.html#extending-sphinx
 myst-parser = "*"
 

docs/api/core_config.md

@@ -0,0 +1,103 @@
+# Shepherd-Core - Config-Models
+
+:::{note}
+TODO: WORK IN PROGRESS
+:::
+
+The models support
+
+- auto-completion with neutral / sensible values
+- complex and custom datatypes (i.e. PositiveInt, lists-checks on length)
+- checking of inputs and type-casting
+- generate their own schema (for web-forms)
+- pre-validation
+- store to & load from yaml with typecheck through wrapper
+- included documentation
+
+## Experiment
+
+This category includes configuration-models for setting up an experiment. Part of the sub-systems are in the next section [](#observer-capabilities).
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.experiment.Experiment
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.experiment.TargetConfig
+```
+
+## Observer Capabilities
+
+These are some of the sub-systems for configuring [experiments](#experiment) and also [tasks](#tasks).
+
+[Link to Source](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/data_models/experiment)
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.experiment.PowerTracing
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.experiment.GpioTracing
+```
+
+```
+deactiveated due to Error (TODO)
+.. autopydantic_model:: shepherd_core.data_models.experiment.GpioActuation
+.. autopydantic_model:: shepherd_core.data_models.experiment.GpioLevel
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.experiment.SystemLogging
+```
+
+## Content-Types
+
+Reusable user-defined meta-data for fw, h5 and vsrc-definitions.
+
+[Link to Source](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/data_models/content)
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.content.EnergyEnvironment
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.content.Firmware
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.content.VirtualHarvesterConfig
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.content.VirtualSourceConfig
+```
+
+## Tasks
+
+These are digestible configs for shepherd-herd or -sheep.
+
+[Link to Source](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/data_models/experiment)
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.task.HarvestTask
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.task.EmulationTask
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.task.FirmwareModTask
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.task.ProgrammingTask
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.task.ObserverTasks
+```
+
+```{eval-rst}
+.. autopydantic_model:: shepherd_core.data_models.task.TestbedTasks
+```

docs/api/core_vsrc.md

@@ -0,0 +1,31 @@
+# Shepherd-Core - Virtual Source Sim
+
+:::{note}
+TODO: WORK IN PROGRESS
+:::
+
+These are simulation-models of the versions running on the PRUs.
+
+## Virtual Source
+
+```{eval-rst}
+.. autoclass:: shepherd_core.vsource.VirtualSourceModel
+    :members:
+    :inherited-members:
+```
+
+## Virtual Converter
+
+```{eval-rst}
+.. autoclass:: shepherd_core.vsource.VirtualConverterModel
+    :members:
+    :inherited-members:
+```
+
+## Virtual Harvester
+
+```{eval-rst}
+.. autoclass:: shepherd_core.vsource.VirtualHarvesterModel
+    :members:
+    :inherited-members:
+```

docs/api/core_waveform_decoder.md

@@ -0,0 +1,13 @@
+# Shepherd-Core - Waveform Decoder
+
+:::{note}
+TODO: WORK IN PROGRESS
+:::
+
+## UART
+
+```{eval-rst}
+.. autoclass:: shepherd_core.decoder_waveform.Uart
+    :members:
+    :inherited-members:
+```

docs/api/data_hdf5.md

@@ -0,0 +1,42 @@
+# Shepherd-Data - HDF5 Reader & Writer
+
+:::{note}
+TODO: WORK IN PROGRESS
+:::
+
+These two classes allow the user to read and write shepherds hdf5-files.
+For more details about the data-format you can read:
+
+- doc for [](../user/data_format.md)
+- [Reader-source](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/reader.py) in core-lib
+- [Reader-source](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_data/shepherd_data/reader.py) in data-module (Inherits from core-reader)
+- [Writer-source](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/writer.py)
+
+## Usage-Example
+
+This basic example can read & print some metadata from a hdf5-file.
+
+```python
+import shepherd_data as sd
+
+with sd.Reader("./hrv_sawtooth_1h.h5") as db:
+    print(f"Mode: {db.get_mode()}")
+    print(f"Window: {db.get_window_samples()}")
+    print(f"Config: {db.get_config()}")
+```
+
+## Reader
+
+```{eval-rst}
+.. autoclass:: shepherd_data.Reader
+    :members:
+    :inherited-members:
+```
+
+## Writer
+
+```{eval-rst}
+.. autoclass:: shepherd_data.Writer
+    :members:
+    :inherited-members:
+```

docs/api/herd.md

@@ -0,0 +1,11 @@
+# Shepherd-Herd
+
+:::{note}
+TODO: WORK IN PROGRESS
+:::
+
+```{eval-rst}
+.. autoclass:: shepherd_herd.Herd
+    :members:
+    :inherited-members:
+```

docs/api/readme.txt

@@ -0,0 +1,19 @@
+extracting Core-API with newer version (>=23.12.1) FAILs with
+
+  File "/home/hans/.local/share/virtualenvs/shepherd-JJj2p24W/lib/python3.10/site-packages/shepherd_core/data_models/content/virtual_source.py", line 44, in Virtua
+lSourceConfig
+    harvester: VirtualHarvesterConfig = VirtualHarvesterConfig(name="mppt_opt")
+  File "/home/hans/.local/share/virtualenvs/shepherd-JJj2p24W/lib/python3.10/site-packages/pydantic/main.py", line 171, in __init__
+    self.__pydantic_validator__.validate_python(data, self_instance=self)
+pydantic_core._pydantic_core.ValidationError: 1 validation error for Config for the Harvester
+  Value error, Component 'virtualharvesterconfig' not found! [type=value_error, input_value={'name': 'mppt_opt'}, input_type=dict]
+
+even when datalib-unittests run through?!?
+
+useful commands:
+
+pip uninstall shepherd-core
+
+git submodule update --recursive --remote
+rm ~/.cache/shepherd_datalib/fixtures.pickle
+pip install ../software/shepherd-datalib/shepherd_core/

docs/api/sheep_high_level.md

@@ -0,0 +1,100 @@
+# Shepherd-Sheep
+
+The shepherd-sheep API offers high level access to shepherd's functionality and forms the base for the two command line utilities.
+With the introduction of the [core-lib](https://pypi.org/project/shepherd-core/) the api was simplified and modernized with a model-based approach. The [pydantic](https://docs.pydantic.dev) data-models offer self-validating config-parameters with neutral defaults.
+
+For lower-level access, have a look at the [](sheep_low_level.md). There is a third option called `debug-api`, used i.e. by the programmer.
+It will not be documented here.
+To learn about the functionality [the source](https://github.com/orgua/shepherd/blob/main/software/python-package/shepherd_sheep/shepherd_emulator.py) should be consulted.
+
+## Harvesting
+
+The `run_harvester()`-function can be used to configure all relevant hardware and software and to sample and extract data from the analog frontend.
+
+```python
+from contextlib import ExitStack
+
+from shepherd_core.data_models.task import HarvestTask
+from shepherd_sheep.shepherd_harvester import ShepherdHarvester
+from shepherd_sheep.logger import set_verbosity
+```
+```{literalinclude} ../../software/python-package/shepherd_sheep/__init__.py
+:language: python
+:pyobject: run_harvester
+```
+
+The snippet is taken from the actual implementation in [sheep/init](https://github.com/orgua/shepherd/blob/main/software/python-package/shepherd_sheep/__init__.py) and references the [HarvestTask](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/data_models/task/harvest.py)
+
+## Emulating
+
+The `run_emulator()`-function can be used to emulate previously recorded IV data for an attached sensor node.
+
+```python
+from contextlib import ExitStack
+
+from shepherd_core.data_models.task import EmulationTask
+from shepherd_sheep.shepherd_emulator import ShepherdEmulator
+from shepherd_sheep.logger import set_verbosity
+```
+```{literalinclude} ../../software/python-package/shepherd_sheep/__init__.py
+:language: python
+:pyobject: run_emulator
+```
+
+The snippet is taken from the actual implementation in [sheep/init](https://github.com/orgua/shepherd/blob/main/software/python-package/shepherd_sheep/__init__.py) and references the [EmulationTask](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/data_models/task/emulation.py).
+
+:::{note}
+TODO: add user/task-config and relink both tasks above
+:::
+
+## Modify Firmware
+
+The `run_firmware_mod()`-function can be used to customize a firmware before flashing it.
+
+```python
+import shutil
+
+from shepherd_core.data_models import FirmwareDType
+from shepherd_core.data_models.task import FirmwareModTask
+from shepherd_core.fw_tools import extract_firmware
+from shepherd_core.fw_tools import firmware_to_hex
+from shepherd_core.fw_tools import modify_uid
+from shepherd_sheep.logger import set_verbosity
+from shepherd_sheep.sysfs_interface import check_sys_access
+```
+```{literalinclude} ../../software/python-package/shepherd_sheep/__init__.py
+:language: python
+:pyobject: run_firmware_mod
+```
+
+The snippet is taken from the actual implementation in [sheep/init](https://github.com/orgua/shepherd/blob/main/software/python-package/shepherd_sheep/__init__.py) and references the [FirmwareModTask](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/data_models/task/firmware_mod.py).
+
+## Program Target
+
+The `run_programmer()`-function can flash a `.hex`-file to a target of choice.
+
+```python
+from contextlib import ExitStack
+
+from shepherd_core.data_models.task import ProgrammingTask
+from shepherd_sheep import sysfs_interface
+from shepherd_sheep.logger import set_verbosity
+from shepherd_sheep.shepherd_debug import ShepherdDebug
+# Note: probably some includes missing
+```
+```{literalinclude} ../../software/python-package/shepherd_sheep/__init__.py
+:language: python
+:pyobject: run_programmer
+```
+
+The snippet is taken from the actual implementation in [sheep/init](https://github.com/orgua/shepherd/blob/main/software/python-package/shepherd_sheep/__init__.py) and references the [ProgrammingTask](https://github.com/orgua/shepherd-datalib/blob/main/shepherd_core/shepherd_core/data_models/task/programming.py).
+
+## Example-Code
+
+This snippet shows the harvester and emulator instantiated with custom config-models. It was used as a 10h stress-test to find a memory leak.
+
+```{literalinclude} ../../software/python-package/tests_manual/testbench_longrun.py
+:language: python
+```
+
+Source: [./tests_manual/testbench_longrun.py](https://github.com/orgua/shepherd/blob/main/software/python-package/tests_manual/testbench_longrun.py)

docs/api/sheep_low_level.md

@@ -0,0 +1,19 @@
+# Shepherd-Sheep - Advanced
+
+This API is more low level than [the approach described here](sheep_high_level).
+
+## Harvester-API
+
+```{eval-rst}
+.. autoclass:: shepherd_sheep.ShepherdHarvester
+    :members:
+    :inherited-members:
+```
+
+## Emulator-API
+
+```{eval-rst}
+.. autoclass:: shepherd_sheep.ShepherdEmulator
+    :members:
+    :inherited-members:
+```

docs/conf.py

@@ -35,7 +35,7 @@
     "sphinx_design",
     "sphinx_copybutton",
     # "sphinxcontrib.typer",
-    # "sphinx.ext.autosectionlabel",
+    "sphinxcontrib.autodoc_pydantic",
 ]
 # TODO: check other sphinx-plugins (mentioned in pipfile) & breathe
 
@@ -49,6 +49,17 @@
 autodoc_mock_imports = ["dbus"]
 
 myst_enable_extensions = ["colon_fence"]
+myst_heading_anchors = 3
+
+autodoc_pydantic_model_show_json = False
+autodoc_pydantic_settings_show_json = False
+autodoc_pydantic_model_show_config_summary = False
+autodoc_pydantic_model_show_validator_summary = False
+autodoc_pydantic_model_show_validator_members = False
+autodoc_pydantic_model_show_field_summary = True
+autodoc_pydantic_model_summary_list_order = "bysource"
+autodoc_pydantic_model_member_order = "bysource"
+autodoc_pydantic_field_list_validators = False
 
 # -- Options for HTML output -------------------------------------------------