Add Spectrogram Visualization to Recordings

.gitattributes

@@ -0,0 +1 @@
+assets/example.wav filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1 @@
+.DS_Store

.pre-commit-config.yaml

@@ -1,22 +1,61 @@
-files: ^(bats_ai)/
+exclude: |
+  (?x)(
+      ^.github/ |
+      ^terraform/ |
+      ^staticfiles/ |
+      ^client/
+  )
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.5.0
     hooks:
       - id: check-case-conflict
       - id: check-docstring-first
+      - id: check-executables-have-shebangs
       - id: check-json
       - id: check-merge-conflict
+      - id: check-shebang-scripts-are-executable
       - id: check-toml
       - id: check-yaml
+        args: ['--unsafe']
       - id: debug-statements
       - id: destroyed-symlinks
       - id: detect-private-key
       - id: double-quote-string-fixer
       - id: end-of-file-fixer
       - id: mixed-line-ending
+        args: ['--fix=lf']
+      - id: pretty-format-json
+        name: Format JSON files
+        args: ["--autofix"]
       - id: trailing-whitespace
 
+  - repo: https://github.com/hadolint/hadolint
+    rev: v2.12.1-beta
+    hooks:
+      - id: hadolint-docker
+        name: Hadolint for Dockerfiles
+
+  - repo: https://github.com/lovesegfault/beautysh
+    rev: v6.2.1
+    hooks:
+      - id: beautysh
+        name: Beautysh for Bash scripts
+        args: ["--indent-size", "4"]
+
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.38.0
+    hooks:
+      - id: markdownlint-fix-docker
+        name: MarkdownLint for Markdown files
+        args: ["--fix"]
+
+  - repo: https://github.com/sirwart/ripsecrets
+    rev: v0.1.7
+    hooks:
+      - id: ripsecrets
+        name: RIPSecrets for API keys and secrets
+
   - repo: https://github.com/psf/black
     rev: 23.9.1
     hooks:

README.md

@@ -1,41 +1,61 @@
 # bats-ai
 
 ## Develop with Docker (recommended quickstart)
+
 This is the simplest configuration for developers to start with.
 
 ### Initial Setup
+
 1. Run `docker compose run --rm django ./manage.py migrate`
 2. Run `docker compose run --rm django ./manage.py createsuperuser`
    and follow the prompts to create your own user
+3. Run  `docker compose run --rm django ./manage.py makeclient \
+                            --username [email protected] \
+                            --uri http://localhost:3000/`
+
+### Run Vue Frontend
+
+1. Run `npm install`
+2. Run `npm run dev`
 
 ### Run Application
+
 1. Run `docker compose up`
-2. Access the site, starting at http://localhost:8000/admin/
+2. Access the site, starting at <http://localhost:8000/admin/>
 3. When finished, use `Ctrl+C`
 
 ### Application Maintenance
+
 Occasionally, new package dependencies or schema changes will necessitate
 maintenance. To non-destructively update your development stack at any time:
+
 1. Run `docker compose pull`
 2. Run `docker compose build --pull --no-cache`
 3. Run `docker compose run --rm django ./manage.py migrate`
-3. Run `docker compose run --rm django ./manage.py createsuperuser`
-4. Run `docker compose run --rm django ./manage.py loaddata species` to load species data into the database
-5. Run  `docker compose run --rm django ./manage.py makeclient --username [email protected] --uri http://localhost:3000/`
+4. Run `docker compose run --rm django ./manage.py createsuperuser`
+5. Run `docker compose run --rm django ./manage.py loaddata species` to load species
+   data into the database
+6. Run  `docker compose run --rm django ./manage.py makeclient \
+                            --username [email protected] \
+                            --uri http://localhost:3000/`
+
 ## Develop Natively (advanced)
+
 This configuration still uses Docker to run attached services in the background,
 but allows developers to run Python code on their native system.
 
 ### Dev Tool Endpoints
+
 1. Main Site Interface [http://localhost:3000/](http://localhost:3000/)
 2. Site Administration [http://localhost:8000/admin/](http://localhost:8000/admin/)
 3. Swagger API (These are default swagger endpoints using Django-REST) [http://localhost:8000/api/docs/swagger/](http://localhost:8000/api/docs/swagger/)
 4. Django Ninja API [http://localhost:8000/api/v1/docs#/](http://localhost:8000/api/v1/docs#/)
-5. MinIO (S3 local management) [http://localhost:9001/browser](http://localhost:9001/browser) 
+5. MinIO (S3 local management) [http://localhost:9001/browser](http://localhost:9001/browser)
    Username: 'minioAccessKey'
    Password: 'minioSecretKey'
 
-### Initial Setup
+### Initial Setup (Natively)
+
 1. Run `docker compose -f ./docker-compose.yml up -d`
 2. Install Python 3.10
 3. Install
@@ -46,8 +66,9 @@ but allows developers to run Python code on their native system.
 7. Run `./manage.py migrate`
 8. Run `./manage.py createsuperuser` and follow the prompts to create your own user
 
-### Run Application
-1.  Ensure `docker compose -f ./docker-compose.yml up -d` is still active
+### Run Application (Natively)
+
+1. Ensure `docker compose -f ./docker-compose.yml up -d` is still active
 2. Run:
    1. `source ./dev/export-env.sh`
    2. `./manage.py runserver`
@@ -58,42 +79,101 @@ but allows developers to run Python code on their native system.
 5. To destroy the stack and start fresh, run `docker compose down -v`
 
 ## Remap Service Ports (optional)
-Attached services may be exposed to the host system via alternative ports. Developers who work
-on multiple software projects concurrently may find this helpful to avoid port conflicts.
 
-To do so, before running any `docker compose` commands, set any of the environment variables:
+Attached services may be exposed to the host system via alternative ports. Developers
+who work on multiple software projects concurrently may find this helpful to avoid
+port conflicts.
+
+To do so, before running any `docker compose` commands, set any of the environment
+variables:
+
 * `DOCKER_POSTGRES_PORT`
 * `DOCKER_RABBITMQ_PORT`
 * `DOCKER_MINIO_PORT`
 
 The Django server must be informed about the changes:
-* When running the "Develop with Docker" configuration, override the environment variables:
+
+* When running the "Develop with Docker" configuration, override the environment
+  variables:
   * `DJANGO_MINIO_STORAGE_ENDPOINT`, using the port from `DOCKER_MINIO_PORT`.
-* When running the "Develop Natively" configuration, override the environment variables:
+* When running the "Develop Natively" configuration, override the environment
+  variables:
   * `DJANGO_DATABASE_URL`, using the port from `DOCKER_POSTGRES_PORT`
   * `DJANGO_CELERY_BROKER_URL`, using the port from `DOCKER_RABBITMQ_PORT`
   * `DJANGO_MINIO_STORAGE_ENDPOINT`, using the port from `DOCKER_MINIO_PORT`
 
-Since most of Django's environment variables contain additional content, use the values from
-the appropriate `dev/.env.docker-compose*` file as a baseline for overrides.
+Since most of Django's environment variables contain additional content, use the
+values from the appropriate `dev/.env.docker-compose*` file as a baseline for
+overrides.
 
 ## Testing
-### Initial Setup
+
+### Initial Setup (Testing)
+
 tox is used to execute all tests.
 tox is installed automatically with the `dev` package extra.
 
-When running the "Develop with Docker" configuration, all tox commands must be run as
-`docker-compose run --rm django tox`; extra arguments may also be appended to this form.
+When running the "Develop with Docker" configuration, all tox commands must be run
+as `docker-compose run --rm django tox`; extra arguments may also be appended to
+this form.
 
 ### Running Tests
+
 Run `tox` to launch the full test suite.
 
 Individual test environments may be selectively run.
 This also allows additional options to be be added.
 Useful sub-commands include:
+
 * `tox -e lint`: Run only the style checks
 * `tox -e type`: Run only the type checks
 * `tox -e test`: Run only the pytest-driven tests
 
 To automatically reformat all code to comply with
 some (but not all) of the style checks, run `tox -e format`.
+
+## Code Formatting
+
+Any contributed code should be compliant with `PEP8`, which is enforced by
+`flake8` via `pre-commit`. It's recommended that you use `pre-commit` to ensure
+linting procedures are run on any commit you make. See the
+[installation instructions](https://pre-commit.com/#install) for your OS/platform
+to install.
+
+After you have the software installed, run `pre-commit install` on the command line.
+Now every time you commit to this project's code base the linter procedures will
+automatically run over the changed files.
+
+To install the linting Python dependencies, run:
+
+```bash
+pip install -r pre-commit
+```
+
+To run pre-commit on files preemptively from the command line use:
+
+```bash
+ git add .
+ pre-commit run
+
+ # or
+
+ pre-commit run --all-files
+```
+
+See `.pre-commit-config.yaml` for a list of configured linters and fixers.
+
+To add `pre-commit` as a Git hook, run the following:
+
+```bash
+pre-commit install
+```
+
+This will run the `pre-commit` update prior to finalizing a local commit.  This
+is preferred because once the commit is created locally, you will need to rebase
+or otherwise rewrite the commit to make adjustments if done after the fact.
+
+Lastly, the GitLab CI/CD infrastructure runs the same `pre-commit` configuration
+on all pipelines for new MRs.  The automated checks in GitLab are optional, but
+it is highly recommended to perform these checks locally prior to pushing new
+commits.

bats_ai/api.py

@@ -1,25 +1,28 @@
 import logging
 
 from ninja import NinjaAPI
-from ninja.security import HttpBearer
 from oauth2_provider.models import AccessToken
 
 from bats_ai.core.views import RecordingRouter, SpeciesRouter
 
 logger = logging.getLogger(__name__)
 
 
-class GlobalAuth(HttpBearer):
-    def authenticate(self, request, token):
-        logger.warning(f'Checking Token: {token}')
-        print(token)
-        logger.warning(AccessToken.objects.get(token=token))
-        if AccessToken.objects.get(token=token):
-            logger.warning('returning token')
-            return token
+def global_auth(request):
+    if request.user.is_anonymous:
+        token = request.headers.get('Authorization', '').replace('Bearer ', '')
+        if len(token) > 0:
+            try:
+                access_token = AccessToken.objects.get(token=token)
+            except AccessToken.DoesNotExist:
+                access_token = None
+            if access_token and access_token.user:
+                if not access_token.user.is_anonymous:
+                    request.user = access_token.user
+    return not request.user.is_anonymous
 
 
-api = NinjaAPI()
+api = NinjaAPI(auth=global_auth)
 
 api.add_router('/recording/', RecordingRouter)
 api.add_router('/species/', SpeciesRouter)

bats_ai/core/admin/__init__.py

@@ -1,9 +1,11 @@
 from .annotations import AnnotationsAdmin
 from .image import ImageAdmin
 from .recording import RecordingAdmin
+from .spectrogram import SpectrogramAdmin
 
 __all__ = [
+    'AnnotationsAdmin',
     'ImageAdmin',
     'RecordingAdmin',
-    'AnnotationsAdmin',
+    'SpectrogramAdmin',
 ]

bats_ai/core/admin/recording.py

@@ -1,6 +1,11 @@
-from django.contrib import admin
+from django.contrib import admin, messages
+from django.db.models import QuerySet
+from django.http import HttpRequest
+from django.urls import reverse
+from django.utils.safestring import mark_safe
 
 from bats_ai.core.models import Recording
+from bats_ai.core.tasks import recording_compute_spectrogram
 
 
 @admin.register(Recording)
@@ -9,6 +14,7 @@ class RecordingAdmin(admin.ModelAdmin):
         'id',
         'name',
         'audio_file',
+        'spectrogram_status',
         'owner',
         'recorded_date',
         'equipment',
@@ -21,6 +27,29 @@ class RecordingAdmin(admin.ModelAdmin):
     # list_select_related = ['owner']
 
     search_fields = ['name']
+    actions = ['compute_spectrograms']
 
     autocomplete_fields = ['owner']
     readonly_fields = ['created', 'modified']
+
+    @admin.display(
+        description='Spectrogram',
+        empty_value='Not computed',
+    )
+    def spectrogram_status(self, recording: Recording):
+        if recording.has_spectrogram:
+            spectrogram = recording.spectrogram
+            href = reverse('admin:core_spectrogram_change', args=(spectrogram.pk,))
+            description = str(spectrogram)
+            link = mark_safe(f'<a href="{href}">{description}</a>')
+            return link
+        return None
+
+    @admin.action(description='Compute Spectrograms')
+    def compute_spectrograms(self, request: HttpRequest, queryset: QuerySet):
+        counter = 0
+        for recording in queryset:
+            if not recording.has_spectrogram:
+                recording_compute_spectrogram.delay(recording.pk)
+                counter += 1
+        self.message_user(request, f'{counter} recordings queued', messages.SUCCESS)

bats_ai/core/admin/spectrogram.py

@@ -0,0 +1,31 @@
+from django.contrib import admin
+
+from bats_ai.core.models import Spectrogram
+
+
+@admin.register(Spectrogram)
+class SpectrogramAdmin(admin.ModelAdmin):
+    list_display = [
+        'pk',
+        'recording',
+        'image_file',
+        'width',
+        'height',
+        'duration',
+        'frequency_min',
+        'frequency_max',
+    ]
+    list_display_links = ['pk', 'recording']
+    list_select_related = True
+    autocomplete_fields = ['recording']
+    readonly_fields = [
+        'recording',
+        'image_file',
+        'created',
+        'modified',
+        'width',
+        'height',
+        'duration',
+        'frequency_min',
+        'frequency_max',
+    ]