doc improvements

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,44 @@
+---
+name: Bug report
+description: Create a report to help us improve
+title: ''
+labels: 'bug'
+assignees: ''
+
+---
+
+If possible choose the correct project:
+
+- [main-repo](https://github.com/orgua/shepherd/issues)
+- [datalib](https://github.com/orgua/shepherd-datalib/issues)
+- [targets](https://github.com/orgua/shepherd-targets/issues)
+- [webapi](https://github.com/orgua/shepherd_webservice/issues)
+
+## Description
+
+A clear and concise description of what the bug is.
+
+## How to reproduce
+
+Steps to reproduce the behavior.
+
+1. Go to '...'
+2. Type '....'
+3. Scroll down to '....'
+4. See error
+
+## Expected behavior
+
+A clear and concise description of what you expected to happen.
+
+## Screenshots & Error Logs
+
+If applicable, add screenshots to help explain your problem.
+
+## Environment
+
+Give some information about your setup.
+
+## Additional Context
+
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,28 @@
+---
+name: Feature request
+description: Suggest an idea for this project
+title: ''
+labels: 'enhancement'
+assignees: ''
+
+---
+
+If possible choose the correct project:
+
+- [main-repo](https://github.com/orgua/shepherd/issues)
+- [datalib](https://github.com/orgua/shepherd-datalib/issues)
+- [targets](https://github.com/orgua/shepherd-targets/issues)
+- [webapi](https://github.com/orgua/shepherd_webservice/issues)
+
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/workflows/publish_herd.yml

@@ -17,13 +17,13 @@ permissions:
   contents: read
 
 jobs:
-  run-quality-assurance:
-    uses: ./.github/workflows/qa_tests.yml
+  run-quality-control:
+    uses: ./.github/workflows/qc_tests.yml
   deploy:
 
     runs-on: ubuntu-latest
     needs:
-      - run-quality-assurance
+      - run-quality-control
     env:
       herd_path: "./software/shepherd-herd"
 

.github/workflows/qa_tests.yml → .github/workflows/qc_tests.yml

@@ -1,4 +1,4 @@
-name: Test for Quality Assurance
+name: Test for Quality Control
 
 on:
   push:

.github/workflows/sphinx_to_pages.yml

@@ -7,12 +7,12 @@ on:
   workflow_call:
 
 jobs:
-  run-quality-assurance:
-    uses: ./.github/workflows/qa_tests.yml
+  run-quality-control:
+    uses: ./.github/workflows/qc_tests.yml
   build-pages:
     runs-on: ubuntu-latest
     needs:
-      - run-quality-assurance
+      - run-quality-control
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}

docs/conf.py

@@ -36,7 +36,7 @@
     "sphinx_design",
     "sphinx_copybutton",
 ]
-# TODO: check other sphinx-pluging (mentioned in pipfile) & breathe
+# TODO: check other sphinx-plugins (mentioned in pipfile) & breathe
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]

docs/external/shepherd_core.md

@@ -1,7 +1,8 @@
 # Core Library
 
 [![PyPiVersion](https://img.shields.io/pypi/v/shepherd_core.svg)](https://pypi.org/project/shepherd_core)
-[![Pytest](https://github.com/orgua/shepherd-datalib/actions/workflows/python-app.yml/badge.svg)](https://github.com/orgua/shepherd-datalib/actions/workflows/python-app.yml)
+[![image](https://img.shields.io/pypi/pyversions/shepherd_core.svg)](https://pypi.python.org/pypi/shepherd-core)
+[![Pytest](https://github.com/orgua/shepherd-datalib/actions/workflows/py_unittest.yml/badge.svg)](https://github.com/orgua/shepherd-datalib/actions/workflows/py_unittest.yml)
 [![CodeStyle](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
 **Documentation**: <https://orgua.github.io/shepherd/external/shepherd_core.html>
@@ -12,7 +13,7 @@
 
 ---
 
-This Python Module is designed as a library and bundles data-models and file-access-routines for the shepherd-testbed, that are used by several codebases.
+The Python Module is designed as a library and bundles data-models and file-access-routines for the shepherd-testbed, that are used by several codebases.
 
 For postprocessing shepherds .h5-files users want to use [shepherd_data](https://pypi.org/project/shepherd_data).
 
@@ -31,6 +32,20 @@ For postprocessing shepherds .h5-files users want to use [shepherd_data](https:/
 
 See [examples](https://github.com/orgua/shepherd-datalib/tree/main/shepherd_core/examples) for more details and usage. Most functionality is showcased there. The [extra](https://github.com/orgua/shepherd-datalib/tree/main/shepherd_core/extra)-directory holds data-generators relevant for the testbed. Notably is a trafficbench-experiment that's used to derive the link-matrix.
 
+### Compatibility
+
+| OS      |   PyVersion  | Comment                                    |
+|---------|--------------|--------------------------------------------|
+| Ubuntu  | 3.8 - 3.12   |                                            |
+| Windows | 3.8 - 3.12   | no support for elf and hex-conversions yet |
+| MacOS   | 3.8 - 3.12   | hex-conversion missing                     |
+
+Notes:
+- hex-conversion needs a working and accessible objcopy
+- elf-supports needs
+  - ``shepherd-core[elf]`` installs ``pwntools-elf-only``
+  - most elf-features also still utilize hex-conversion
+
 ### Data-Models in Detail
 
 - new orchestration ``/data-models`` with focus on remote shepherd-testbed

docs/external/shepherd_data.md

@@ -1,7 +1,8 @@
 # Data Module
 
 [![PyPiVersion](https://img.shields.io/pypi/v/shepherd_data.svg)](https://pypi.org/project/shepherd_data)
-[![Pytest](https://github.com/orgua/shepherd-datalib/actions/workflows/python-app.yml/badge.svg)](https://github.com/orgua/shepherd-datalib/actions/workflows/python-app.yml)
+[![image](https://img.shields.io/pypi/pyversions/shepherd_data.svg)](https://pypi.python.org/pypi/shepherd-data)
+[![Pytest](https://github.com/orgua/shepherd-datalib/actions/workflows/py_unittest.yml/badge.svg)](https://github.com/orgua/shepherd-datalib/actions/workflows/py_unittest.yml)
 [![CodeStyle](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
 **Documentation**: <https://orgua.github.io/shepherd/external/shepherd_data.html>
@@ -47,7 +48,7 @@ with sd.Reader("./hrv_sawtooth_1h.h5") as db:
   - file can be checked for plausibility and validity (`is_valid()`)
   - internal structure of h5file (`get_metadata()` or `save_metadata()` ... to yaml) with lots of additional data
   - access data and various converters, calculators
-    - `read_buffers()` -> generator that provides one buffer per call, can be configured on first call
+    - `read_buffers()` ⇾ generator that provides one buffer per call, can be configured on first call
     - `get_calibration_data()`
     - `get_windows_samples()`
     - `get_mode()`
@@ -91,7 +92,6 @@ with sd.Reader("./hrv_sawtooth_1h.h5") as db:
   - `is_valid`
   - `energy()`
   - `check_timediffs()`
-  - `data_timediffs()`
   - `get_metadata()`
   - `save_metadata()`
 
@@ -205,20 +205,20 @@ TODO:
   - note: lzf seems to cause trouble with some third party hdf5-tools
   - compression is a heavy load for the beaglebone, but it got more performant with recent python-versions
 - size-experiment A: 24 h of ramping / sawtooth (data is repetitive with 1 minute ramp)
-  - gzip-1: 49'646 MiB -> 588 KiB/s
-  - lzf: 106'445 MiB -> 1262 KiB/s
-  - uncompressed: 131'928 MiB -> 1564 KiB/s
+  - gzip-1: 49'646 MiB ⇾ 588 KiB/s
+  - lzf: 106'445 MiB ⇾ 1262 KiB/s
+  - uncompressed: 131'928 MiB ⇾ 1564 KiB/s
 - cpu-load-experiments (input is 24h sawtooth, python 3.10 with most recent libs as of 2022-04)
   - warning: gpio-traffic and other logging-data can cause lots of load
 
 ```
-  emu_120s_gz1_to_gz1.h5 	-> emulator, cpu_util [%] = 65.59, data-rate =  352.0 KiB/s
-  emu_120s_gz1_to_lzf.h5 	-> emulator, cpu_util [%] = 57.37, data-rate =  686.0 KiB/s
-  emu_120s_gz1_to_unc.h5 	-> emulator, cpu_util [%] = 53.63, data-rate = 1564.0 KiB/s
-  emu_120s_lzf_to_gz1.h5 	-> emulator, cpu_util [%] = 63.18, data-rate =  352.0 KiB/s
-  emu_120s_lzf_to_lzf.h5 	-> emulator, cpu_util [%] = 58.60, data-rate =  686.0 KiB/s
-  emu_120s_lzf_to_unc.h5 	-> emulator, cpu_util [%] = 55.75, data-rate = 1564.0 KiB/s
-  emu_120s_unc_to_gz1.h5 	-> emulator, cpu_util [%] = 63.84, data-rate =  351.0 KiB/s
-  emu_120s_unc_to_lzf.h5 	-> emulator, cpu_util [%] = 57.28, data-rate =  686.0 KiB/s
-  emu_120s_unc_to_unc.h5 	-> emulator, cpu_util [%] = 51.69, data-rate = 1564.0 KiB/s
+emu_120s_gz1_to_gz1.h5 	⇾ emulator, cpu_util [%] = 65.59, data-rate =  352.0 KiB/s
+emu_120s_gz1_to_lzf.h5 	⇾ emulator, cpu_util [%] = 57.37, data-rate =  686.0 KiB/s
+emu_120s_gz1_to_unc.h5 	⇾ emulator, cpu_util [%] = 53.63, data-rate = 1564.0 KiB/s
+emu_120s_lzf_to_gz1.h5 	⇾ emulator, cpu_util [%] = 63.18, data-rate =  352.0 KiB/s
+emu_120s_lzf_to_lzf.h5 	⇾ emulator, cpu_util [%] = 58.60, data-rate =  686.0 KiB/s
+emu_120s_lzf_to_unc.h5 	⇾ emulator, cpu_util [%] = 55.75, data-rate = 1564.0 KiB/s
+emu_120s_unc_to_gz1.h5 	⇾ emulator, cpu_util [%] = 63.84, data-rate =  351.0 KiB/s
+emu_120s_unc_to_lzf.h5 	⇾ emulator, cpu_util [%] = 57.28, data-rate =  686.0 KiB/s
+emu_120s_unc_to_unc.h5 	⇾ emulator, cpu_util [%] = 51.69, data-rate = 1564.0 KiB/s
 ```

docs/external/shepherd_targets.md

@@ -12,7 +12,8 @@
 
 ---
 
-The repositoty contains a collection of target-hardware and mcu-firmwares used by the shepherd-testbed. In the past shepherd had a nRF52-only target but can now leverage an additional FRAM, as demonstrated successfully by the [Riotee-Platform](https://www.riotee.nessie-circuits.de).
+The repository contains a collection of target-hardware and mcu-firmwares used by the shepherd-testbed. In the past shepherd had a nRF52-only target but can now leverage an additional FRAM, as demonstrated successfully by the 
+[Riotee-Platform](https://www.riotee.nessie-circuits.de).
 
 **NOTE**: The shepherd testbed has a second target-port (unused) and is looking for ideas. Just contact us.
 

docs/external/shepherd_webservice.md

@@ -2,6 +2,8 @@
 
 [![QA-Tests](https://github.com/orgua/shepherd_webservice/actions/workflows/qa_tests.yml/badge.svg)](https://github.com/orgua/shepherd_webservice/actions/workflows/qa_tests.yml)
 
+**Testbed-WebAPI**: <https://shepherd.cfaed.tu-dresden.de:8000>
+
 **Documentation**: <https://orgua.github.io/shepherd/external/shepherd_webservice.html>
 
 **Source Code**: <https://github.com/orgua/shepherd-webservice>
@@ -10,11 +12,11 @@
 
 ---
 
-The Webservice links the user to the testbed. It's written in Python and uses FastAPI to expose an interface to the internet. You can write your own tools or just use the tb-client integrated in the [Core-Datalib](https://pypi.org/project/shepherd_core).
+The Webservice links the user to the testbed. It's written in Python and uses FastAPI to expose an interface to the internet. You can write your own tools or just use the testbed-client integrated in the [Core-Datalib](https://pypi.org/project/shepherd_core).
 
-Internally the webservices utilizes the [herd-lib](https://pypi.org/project/shepherd_herd) to access the shepherd observers. Handling of data is done with Pydantic-Models that store the actual data a database.
+Internally the webservices utilizes the [herd-lib](https://pypi.org/project/shepherd_herd) to access the shepherd observers. Handling of data is done with Pydantic-Models that store the actual data in a database.
 
-## FastApi Webservice
+## FastApi Webservice (current prototype)
 
 - DNS approved for the server: shepherd.cfaed.tu-dresden.de
 - Port 8000 requested for firewall passing
@@ -23,9 +25,90 @@ Internally the webservices utilizes the [herd-lib](https://pypi.org/project/shep
   - add [https to FastAPI](https://fastapi.tiangolo.com/deployment/https/)
 - TODO: allow service to start with reduced rights -> get nfs in order (access needs elevation)
 
+[API-Website](http://127.0.0.1:8000/)
+[ReDoc](http://127.0.0.1:8000/doc)
+[OpenApiDoc](http://127.0.0.1:8000/doc0)
+
+
 ----
 
-**TODO**
+## Cornerstones
+
+### Data-Containers
+
+[Pydantic](https://github.com/pydantic/pydantic)
+
+- data validation for python dataclasses, its fast, elegantly designed and comes with batteries included
+- already trusted base for [shepherd-datalib](https://github.com/orgua/shepherd-datalib)
+
+### WebFrontend
+
+[FastUI](https://github.com/pydantic/FastUI)
+
+- restful web-framework based on pydantic2 & fastapi
+- still alpha, but usable, rapid updates
+- plan B: tbd
+
+[Streamlit](https://streamlit.io)
+- stable and lots to offer, but maybe limited for more
+- and there is a freemium service
+
+### WebApi
+
+[FastApi](https://fastapi.tiangolo.com/)
+- high performance web framework for APIs based on pydantic
+- offers features like OAuth, SSL-Encryption (by uvicorn), user-sessions
+- **choice for now**
+
+[Flask](https://flask.palletsprojects.com/en/3.0.x/)
+- low level but stable framework
+
+### Database
+
+[Beanie](https://github.com/roman-right/beanie)
+- async & pydantic-based ODM or MongoDB
+- stable, but also actively developed
+- **choice for now**
+
+[SQLModel](https://github.com/tiangolo/sqlmodel)
+- async & pydantic-based ODM for SQLAlchemy
+- still alpha, very slow in development
+- has trouble with complex pydantic-models
+
+[ORMDantic](https://github.com/yezz123/ormdantic)
+- async & pydantic-based ODM for SQLAlchemy
+- currently limited to pydantic <v2 -> dealbreaker
+
+### Misc
+
+- Secrets-Management
+  - https://pypi.org/project/python-secrets/
+- Server-Monitoring, remote alerting
+  - sentry-sdk,
+
+
+## install
+
+- set up a local MongoDB instance
+- install package
+- config .env, by either bringing in a backup or starting fresh
+  - backup: repopulate database by using ´shepherd_wsrv init file´
+  - fresh start: generate [fresh salt](https://github.com/orgua/shepherd_webservice/blob/main/scripts/salt_generator.py) and initialize database with `shepherd_wsrv init`
+
+## run server
+
+```Shell
+shepherd_wsrv run
+```
+
+or to switch to the offline-mode activate the redirect to the docs
+
+```Shell
+shepherd_wsrv redirect
+```
+
+## save state
 
-- bring out of alpha-stage
-- show structure, relations between sw-components
+```Shell
+shepherd_wsrv backup file_name
+```

docs/index.rst

@@ -9,7 +9,9 @@ Welcome to SHEPHERD's documentation
 To learn how *shepherd* enables research into the most challenging problems of coordinating battery-less sensor nodes, take a look at our `paper <https://wwwpub.zih.tu-dresden.de/~mzimmerl/pubs/geissdoerfer19shepherd.pdf>`_.
 To get a basic understanding of what shepherd does, read the :doc:`user/basics`.
 If you have the hardware on your desk and want to get started, read :doc:`user/getting_started`.
-To record/emulate data on a group of shepherd nodes, use the :ref:`shepherd-herd-cli` command line utility.
+To record/emulate data on a group of shepherd nodes, use the :ref:`herd-cli` command line utility.
+To access the testbed-instance go to :doc:`external/testbed`.
+The DevLog-Documentation is in: https://orgua.github.io/shepherd_v2_planning/
 
 .. toctree::
    :maxdepth: 2
@@ -29,11 +31,11 @@ To record/emulate data on a group of shepherd nodes, use the :ref:`shepherd-herd
    :maxdepth: 2
    :caption: Testbed & Tools
 
-   external/testbed.md
-   external/shepherd_core.md
-   external/shepherd_data.md
-   external/shepherd_targets.md
-   external/shepherd_webservice.md
+   external/testbed
+   external/shepherd_core
+   external/shepherd_data
+   external/shepherd_targets
+   external/shepherd_webservice
 
 .. toctree::
    :maxdepth: 2
@@ -50,7 +52,7 @@ To record/emulate data on a group of shepherd nodes, use the :ref:`shepherd-herd
    :maxdepth: 2
    :caption: Analyzing Sync-Behavior
 
-   timesync/1_prepare_software.md
-   timesync/2_setup_hardware.md
-   timesync/3_measurement.md
-   timesync/4_analysis.md
+   timesync/1_prepare_software
+   timesync/2_setup_hardware
+   timesync/3_measurement
+   timesync/4_analysis

docs/requirements.txt

@@ -1,5 +1,3 @@
--e ./software/python-package
--e ./software/shepherd-herd
 alabaster
 asn1crypto
 Babel