Migrate to the picamera2 library for camera control & image acquisiti…

…on (#19)

* Add the new imager for picamera2

* work on the imagernew process and the picamera class

* Manage the camera parameters in picamera

* Capture high resolution images while video recording

* Add picamera modifications to the adafruit hat

* Create the exposure time / shutter speed methods

* Update for the purpose of resolving streaming issue

* Draft the picamera2 thread

* Implement the picamera thread

* Replicate the remaining imager process methods for picamera

* Add some newly required packages to `pyproject.toml`

* Fix erroneous separation of camera frame production & consumption across processes

* Merge my adafruithat changes with the pscopehat version

* Try to set up autoformatting for imagernew

* Add linting of imagernew with pylama

* Increase max line length to 100, start refactoring `__message_settings`

* Decompose the `__message_settings` method

* Add import order autoformatting/linting

* Make black inclusions/exclusions consistent with those for isort & pylama

* Add more lint checks, including no non-module `from` imports

* Update GitHub Actions CI job

* Install apt dependency for backend

* Resolve some easier linter complaints

* Fix `poetry install` on the RPi

* Refactor the MJPEG-related classes

* Confirm image saving during preview; comment out camera controls code for now

* Remove `picam_threading`, restore `ImagerProcess` functionality

* Try to unbreak numpy installation on GitHub Actions

* Simplify the files I've rewritten so far

* Update some comments, adjust method names for consistency

* Merge the `streams` file into the `camera` module

* Reorder classes in `camera` module

* Make type checks stricter; fix exposure time controls

* Make white balance gain & exposure time settings work again

* Split the stop-flow domain logic out from the imager module

* Suppress lint error which can only be solved on python 3.10

* Decompose MQTT API for camera settings to separate class

* Move camera controler to its own subpackage

* Improve naming, update TODOs & FIXMEs

* Rewrite camera settings adjustment

* Resolve linter complaints

* Apply initial camera controls settings before starting the camera

* Slightly reorganize hardware config loading

* Update `CHANGELOG.md`

* Tweak variable names and comments

* Add error-handling for invalid camera settings

* Handle uncaught errors in the camera MQTT worker's event loop

* Gracefully handle a missing camera at startup

* Update `CHANGELOG.md`

* Try to fix `poetry install` problems on the Pi with armhf

* Try again to fix `poetry install` problems on armhf Pi

* Update GitHub Actions workflow

* Try to repair GitHub Actions workflow

* Revert my addition of image index as prefix to raw image filenames

* Set JPEG capture quality to the max value

* Fix error in the pscopehat version of the dashboard

* Fix error in reporting of camera name for the IMX477 sensor

---------

Co-authored-by: Ethan Li <lietk12@gmail.com>

.github/workflows/control-build.yml

@@ -0,0 +1,47 @@
+name: Build the hardware controller
+
+on:
+  push:
+    branches:
+      - 'main'
+      - 'beta'
+      - 'stable'
+    tags:
+      - 'control/v*'
+  pull_request:
+  merge_group:
+  workflow_dispatch:
+    inputs:
+      git-ref:
+        description: 'Git ref (optional)'
+        required: false
+
+env:
+  REGISTRY_IMAGE: ghcr.io/planktoscope/device-backend-control
+
+jobs:
+  ci:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install poetry
+        run: pipx install poetry==1.7.1
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+          cache: 'poetry'
+          cache-dependency-path: |
+            control/poetry.lock
+
+      - name: Install build dependencies
+        run: |
+          sudo apt-get install -y libcap-dev
+          poetry -C ./control/ install --with dev,hw-dev --no-root
+
+      - name: Run checks
+        run: poetry -C ./control/ run poe --root ./control/ check

CHANGELOG.md

@@ -8,12 +8,20 @@ All dates in this file are given in the [UTC time zone](https://en.wikipedia.org
 
 ## Unreleased
 
+### Added
+
+- (Hardware controller) A new picamera2-based camera-management module (`camera`) is now available as an alternative to the camera-management part of the previous raspimjpeg-based image-acquisition-and-camera-management module.
+- (Hardware controller) A new image-acquisition module (`imagernew`) is now available for use with the new picamera2-based `camera` module, as an alternative to the image-acquisition part of the previous raspimjpeg-based image-acquisition-and-camera-management module.
+
 ### Changed
 
+- (Hardware controller) The new picamera2-based image-acquisition module (`imagernew`) is now used by default, instead of the previous raspimjpeg-based `imager` module.
 - (Breaking change; segmenter) EcoTaxa export archive filenames are now saved as `ecotaxa_{acquisition id}.zip` instead of `ecotaxa_{project id}_{date}_{sample id}.zip`, which was long and redundant and (because many devices have incorrect system times) inappropriate for viewing files in a logically sorted order.
 
 ### Fixed
 
+- (Breaking change; hardware controller) Images acquired by the hardware controller using the newly-default `imagernew` image-acquisition module now have more unique filenames (which include an incrementing index and the date of image capture, rather than just the time of the image capture).
+- (Hardware controller) The hardware controller using the newly-default `imagernew` image-acquisition module no longer crashes when invalid values are given for camera settings (e.g. null or non-numeric white balance gains).
 - (Hardware controller) The pixel calibration values have been switched between the default v2.5 hardware config file and the default v2.6 hardware config file, so that each file has the correct pixel calibration. The default pscopehat hardware config file has also been updated to include the changes made to the default v2.6 hardware config file.
 - (Breaking change; segmenter) The segmenter now runs as `root` (instead of `pi`) in the Docker container for it, so that it doesn't break on various actual & potential edge cases of files/directories being created with `root` ownership (rather than `pi` ownership) before being bind mounted into the container.
 

README.md

@@ -8,26 +8,17 @@ Note: this is a work-in-progress refactor and is not yet ready for general use.
 
 This repository contains the PlanktoScope's backend software services implementing the PlanktoScope's hardware support, application logic, and public API. The current backend functionality can be divided into two categories:
 
-- Device control: controls the PlanktoScope's sensors and actuators, producing raw data. Currently, device control consists of:
+- [Device control](./control/README.md): controls the PlanktoScope's sensors and actuators, producing raw data. Currently, device control consists of:
   - Hardware abstraction layers: implements a simple, mostly-uniform internal API on different underlying hardware devices, for the application-level hardware controller to be used with different hardware implementations of PlanktoScope designs.
   - Application-level controller: provides device behaviors (such as stop-flow imaging) generalizable across different generations of PlanktoScopes. In software engineering jargon, this is the "business logic" of the PlanktoScope hardware-control software.
   - API adapter: exposes the application-level controller with a standard public API for other software clients to use.
 - Data processing: transforms raw data into processed data for downstream analysis and quantification. Currently, data processing functionality consists of:
-  - Object isolation: detects objects from camera frames and creates an image for each isolated object, with the background removed.
-
-However, the organization of the source code does not yet reflect the organization of software functionalities.
+  - [Object isolation and segmentation](./processing/segmenter/README.md): detects objects from camera frames and creates an image for each isolated object, with the background removed.
 
 ## Usage
 
 TBD
 
-### Prerequisites
-
-To use this project, you'll need:
-
-- Python 3.9
-- Poetry 1.4.2
-
 ## Licensing
 
 Except where otherwise indicated, source code provided here is covered by the following information:

control/.flake8

@@ -0,0 +1,29 @@
+[flake8]
+max-line-length = 100
+imr220_include = *
+# We really want IMR241, but the implementation is buggy (false positives everywhere).
+imr241_include = ''
+exclude =
+  adafruithat/main.py,
+  adafruithat/planktoscope/__init__.py,
+  adafruithat/planktoscope/display.py,
+  adafruithat/planktoscope/identity.py,
+  adafruithat/planktoscope/integrity.py,
+  adafruithat/planktoscope/light.py,
+  adafruithat/planktoscope/mqtt.py,
+  adafruithat/planktoscope/stepper.py,
+  adafruithat/planktoscope/uuidName.py,
+  adafruithat/planktoscope/imager,
+  pscopehat/main.py,
+  pscopehat/main_thread.py,
+  pscopehat/shush,
+  pscopehat/planktoscope/__init__.py,
+  pscopehat/planktoscope/display.py,
+  pscopehat/planktoscope/identity.py,
+  pscopehat/planktoscope/integrity.py,
+  pscopehat/planktoscope/light.py,
+  pscopehat/planktoscope/module.py,
+  pscopehat/planktoscope/mqtt.py,
+  pscopehat/planktoscope/stepper.py,
+  pscopehat/planktoscope/uuidName.py,
+  pscopehat/planktoscope/imager,

control/README.md

@@ -0,0 +1,54 @@
+# control
+
+The PlanktoScope's hardware controller.
+
+Note: this is a work-in-progress refactor and is not yet ready for general use.
+
+## Introduction
+
+This repository contains the PlanktoScope's hardware controller, which controls the PlanktoScope's sensors and actuators, producing raw data. Currently, device control consists of:
+- Hardware abstraction layers: implements a simple, mostly-uniform internal API on different underlying hardware devices, for the application-level hardware controller to be used with different hardware implementations of PlanktoScope designs.
+- Application-level controller: provides device behaviors (such as stop-flow imaging) generalizable across different generations of PlanktoScopes. In software engineering jargon, this is the "business logic" of the PlanktoScope hardware-control software.
+- API adapter: exposes the application-level controller with a standard public API for other software clients to use.
+
+However, the organization of the source code does not yet reflect the organization of software functionalities.
+
+## Usage
+
+### Deployment
+
+Currently the hardware controller can only be deployed as part of the distro setup scripts at [github.com/PlanktoScope/PlanktoScope](https://github.com/PlanktoScope/PlanktoScope).
+
+### Development
+
+To install all dependencies including development tooling, run:
+```
+poetry install --with dev,hw-dev --no-root
+```
+
+Then you can run the code auto-formatter on the project by running:
+```
+poetry run poe fmt
+```
+
+And you can run all checks (including code formatting and linting) by running:
+```
+poetry run poe check
+```
+
+### Prerequisites
+
+To use this project, you'll need:
+
+- Python 3.9
+- Poetry 1.4.2
+
+## Licensing
+
+Except where otherwise indicated, source code provided here is covered by the following information:
+
+Copyright PlanktoScope project contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+You can use the source code provided here under the [GPL 3.0 License](https://www.gnu.org/licenses/gpl-3.0.en.html).

control/adafruithat/main.py

@@ -25,11 +25,11 @@
 
 import planktoscope.mqtt
 import planktoscope.stepper
-import planktoscope.imager
 import planktoscope.light # Fan HAT LEDs
 import planktoscope.identity
 import planktoscope.uuidName # Note: this is deprecated.
 import planktoscope.display # Fan HAT OLED screen
+from planktoscope.imagernew import mqtt as imagernew
 
 # enqueue=True is necessary so we can log accross modules
 # rotation happens everyday at 01:00 if not restarted
@@ -113,9 +113,9 @@ def handler_stop_signals(signum, _):
     # Starts the imager control process
     logger.info("Starting the imager control process (step 3/4)")
     try:
-        imager_thread = planktoscope.imager.ImagerProcess(shutdown_event)
-    except:
-        logger.error("The imager control process could not be started")
+        imager_thread = imagernew.Worker(shutdown_event)
+    except Exception as e:
+        logger.error(f"The imager control process could not be started: {e}")
         imager_thread = None
     else:
         imager_thread.start()

control/adafruithat/planktoscope/camera/__init__.py

@@ -0,0 +1 @@
+"""camera provides low-level functionality for camera control."""