Merge branch 'main' into container-action

.gitattributes

@@ -0,0 +1,2 @@
+assets/example.wav filter=lfs diff=lfs merge=lfs -text
+assets/model.mobilenet.onnx filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,11 @@
+.DS_Store
+/**/*.shp
+/**/*.shx
+/**/*.csv
+models/datasets/
+models/spectrograms/
+models/ignore/
+models/*.jpg
+models/*.pkl
+temp*.jpg
+temp*.png

.pre-commit-config.yaml

@@ -1,22 +1,61 @@
-files: ^(bats_ai)/
+exclude: |
+  (?x)(
+      ^.github/ |
+      ^terraform/ |
+      ^staticfiles/ |
+      ^client/
+  )
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.5.0
     hooks:
       - id: check-case-conflict
       - id: check-docstring-first
+      - id: check-executables-have-shebangs
       - id: check-json
       - id: check-merge-conflict
+      - id: check-shebang-scripts-are-executable
       - id: check-toml
       - id: check-yaml
+        args: ['--unsafe']
       - id: debug-statements
       - id: destroyed-symlinks
       - id: detect-private-key
       - id: double-quote-string-fixer
       - id: end-of-file-fixer
       - id: mixed-line-ending
+        args: ['--fix=lf']
+      - id: pretty-format-json
+        name: Format JSON files
+        args: ["--autofix"]
       - id: trailing-whitespace
 
+  - repo: https://github.com/hadolint/hadolint
+    rev: v2.12.1-beta
+    hooks:
+      - id: hadolint-docker
+        name: Hadolint for Dockerfiles
+
+  - repo: https://github.com/lovesegfault/beautysh
+    rev: v6.2.1
+    hooks:
+      - id: beautysh
+        name: Beautysh for Bash scripts
+        args: ["--indent-size", "4"]
+
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.38.0
+    hooks:
+      - id: markdownlint-fix-docker
+        name: MarkdownLint for Markdown files
+        args: ["--fix"]
+
+  - repo: https://github.com/sirwart/ripsecrets
+    rev: v0.1.7
+    hooks:
+      - id: ripsecrets
+        name: RIPSecrets for API keys and secrets
+
   - repo: https://github.com/psf/black
     rev: 23.9.1
     hooks:

DEPLOYMENT.md

@@ -0,0 +1,91 @@
+# bats-ai
+
+## Deployment with Docker (recommended quickstart)
+
+This was a bit rushed so the deployment utilizes a single
+docker file `docker-compose.prod.yml` in the root of the directory
+
+I wanted some simple instructions below to configure the deployment
+
+Be sure to use the proper hostname (batdetectai.kitware.com) in
+all locations that require it.
+
+## Docker Compose Differences
+
+I created a `client` service which has it's own Dockerfile and
+builds the vue client app.
+The `client` service also uses a reverse proxy to route
+`/api`, `/admin` fields to the django server.
+The client will need to be built with a different Client ID
+for accessing the server.
+
+### Initial Setup for Deployment
+
+1. Run `docker compose run --rm django ./manage.py migrate`
+2. Run `docker compose run --rm django ./manage.py createsuperuser`
+   and follow the prompts to create your own user
+3. Run  `docker compose run --rm django ./manage.py makeclient \
+                            --username [email protected] \
+                            --uri https://batdetectai.kitware.com/`
+4. Run `docker compose run --rm django ./manage.py loaddata species` to load species
+   data into the database
+5. Run `docker compose run --rm django ./manage.py collectstatic`
+   to collect the static files
+6. Run `docker compose -f docker-compose.prod.yml up` to start the server
+   add `-d` for a silent version to run in the background
+7. Copy over the ./dev/.env.prod.docker-compose.template
+   to `./dev/.env.prod.docker-compose.template` and change the default passwords
+8. Change the ID in the `./client/env.production` to a custom ID - this will
+   probably require a `docker compose build` to build the app afterwards
+9. After creating the basic application log into the django admin `batdetectai.kitware.com/admin`
+   and change the ApplicationId to the ID in the `./client.env.production`
+10. Test logging in/out and uploading data to the server.
+
+### GRTS Cell Id suppoer
+
+Make sure that there is the grts.csv in the /opt/batai/dev/grtsCells folder
+
+Then run `docker compose run --rm django ./manage.py importGRTSCells /app/csv/grts.csv`
+
+It may take a few minutes to upload because it is loading
+around 500k rows into the DB.
+
+### system.d service
+
+Service that will automatically start and launch the server
+Create this at `/etc/systemd/system` using sudo
+
+```systemd
+[Unit]
+Description=batai-server
+Requires=docker.service
+After=docker.service
+
+[Service]
+ExecStartPre=/bin/sleep 10
+Environment=PATH=/usr/bin:/sbin:/usr/sbin:/usr/local/sbin:/usr/local/bin
+Restart=always
+User=bryon
+Group=docker
+TimeoutStartSec=300
+RestartSec=20
+WorkingDirectory=/opt/batai
+# Shutdown container (if running) when unit is started
+ExecStartPre=docker compose down
+# Start container when unit is started
+ExecStart=docker compose -f docker-compose.prod.yml up
+# Stop container when unit is stopped
+ExecStop=docker compose down
+
+[Install]
+WantedBy=multi-user.target
+```
+
+After run `sudo systemctl enable batai.service`
+Then to start you can use `sudo systemctl start batai.service`
+Stopping: `sudo systemctl stop batai.service`
+
+### User Management
+
+There is no email server connected up so users need to be
+individually approved and their email verified by an admin

INSTRUCTIONS.md

@@ -0,0 +1,78 @@
+# Instructions
+
+The following are instructions on how to use this application to share and
+annotate bat recordings.
+
+## Uploading Data
+
+When you first login, got to the 'Recordings' tab to view two lists of recordings.
+The first list is a list of recordings you have uploaded.
+You can add new data by clicking the 'Upload+' button and choosing a local
+file on your system.
+A date and name are required for uploading new data.
+Optional data can be added including location/ equipment and comments about the file
+
+**Public** - using this checkbox will share the data with all other users
+within the system
+
+Once others have have made annotations on your own files that are public
+you can view them by clicking on the name
+
+## Others Shared Data
+
+This second list of recording files include files that were made public
+by other users or the system admin
+
+This will allow you create annotations on other's files by
+clicking on the **Name** field for the annotation file
+
+## Annotation Editor
+
+### Viewing Annotations
+
+The annotation editor has two main views of the spectrogram from the recording:
+
+**Main View** - This main view has the full annotation zoomed in and can be
+dragged/panned using left click and can be zoomed using the mousewheel.
+At the top of the screen are the current frequence and time for the mouse cursor.
+On the upper right side of the area are buttons for adding/removing information
+
+**Bat Icon** - turns on/off species annotations for any bounding boxes on the screen
+**MS Icon** - will toggle millisecond text annotations to all boxes
+**KHZ Icon** - will toggle frequency labels for each bounding box on the screen
+**Grid Icon** - toggles gride display across the spectrogram
+**Compressed Icon** - toggles on/off the compressed view for the spectrogram
+
+### Interactions
+
+**Clicking** - clicking inside of annotation will automatically select it.
+It will become cyan in color and the annotation will be selected in either
+the Sequence or Pulse list.
+
+**Right Clicking** - Right clicking on an annotation will swap it into 'Edit Mode'
+In 'Edit Mode' the annotation bounds can be modified by clicking on the corners
+and dragging them around.
+
+#### Full Spectrogram View
+
+Below the main view is a thumbnail of the full spectrogram.  This view shows
+the entire spectrogram.
+The **yellow** bounding box is used to show the current location and zoom
+level for the **Main View**.
+Clicking on and dragging in the thumbnail view will pan and jump instanlty to
+that area in the main view.
+
+#### Sequence and Pulse Annotations
+
+On the right side of the screen is a list of the Sequence and Pulse annotations.
+These are tabbed views that can be switched between by clicking on the Tab Name.
+Annotations can be selected by clicking on them.
+When an annotation is selected it can be edited including the
+species comments and other information.
+
+**Sequence** - A Sequence annotation is typically used to group multiple pulses together
+This can be drawn by clicking on the 'Add+' button and drawing a box around the pulses.
+Once created it is shown at the top of the screen
+
+**Pulse** - A Pulse annotation is an annotation around a single pulse in the system.
+These have a fequency range as well a time range.