Modularization and shared analysis server

[srd424]

Just to let every know, in addition to playing around generating a docker image, I've done the early beginnings of some work on 'modularizing':

https://github.com/srd424/BirdNET-Pi/tree/modules

I know other people have gone down the road of full rewrites, etc., but realistically I have limited time / energy so I'm keeping it simple. I also think it's easier to 'trust' a smaller set of changes to the codebase.

I guess this would eventually form the basis for generating a number of small(er) docker images for different components. The source code itself is pretty small so there seems no point in splitting it up, therefore I've just focused on dependencies (debs and pip packages) for the moment, and only enabling the appropriate services.

The idea is you define MODULES_ENABLED at install time either in birdnet.conf or birdnet.conf.override. Currently recognised modules are server for the analysis server, main for basically everything else, and local_recording which pulls in pulseaudio which is otherwise omitted to save space.

The modules branch also includes some changes to server.py to allow it to handle requests from multiple different 'sites'. This is a bit rudimentary at the moment, you need to bind mount each site's "state" directory on /home/pi/sites/<sitename>, and it's recordings directory on /home/pi/BirdSongs/<sitename>, and it doesn't yet look at the site-specific version of include/exclude lists etc. This also needs some tiny changes to the client which I was thinking of putting into a PR as they're simple, don't really affect anything, and would save me carrying the changes..

Anyway, I've probably got this to the state where this is usable for my own needs (feeding audio from relatives houses to be centrally processed and hosted), I thought I'd make a public post linking to the state of play just in case I don't get time to take it further!

[srd424]

First pass at a working analysis server docker container: https://github.com/srd424/birdnetpi-docker/pkgs/container/bnpi-server

Command line to run would look like something like:

podman run -d --replace -p 5050:5050 --name nuc-bnpi-server -v /state/bnpi-server:/state \
    -v /state/birdnetA:/home/pi/sites/siteA -v /vol/birdrec:/home/pi/BirdSongs/siteA ghcr.io/srd424/bnpi-server:amd64

As ever I'm now "testing this in prod" again on my live server :)

[srd424]

https://github.com/srd424/birdnetpi-docker/pkgs/container/bnpi-site-main is the other side of the coin - the main BNPI website bits without the analysis server (which reduces the image size a bit.) Usage the same as the combined docker image I posted first.

[srd424]

I've managed to improve these containers to use a shared layer containing most of the pip packages that are common between the analysis server and the rest of site:

$ podman history bnpi-server | grep -v '0 B'
ID            CREATED         CREATED BY                                     SIZE        COMMENT
d03ed0253ea3  11 minutes ago  |5 MODULES=server PYTHONDONTWRITEBYTECODE=...  17 MB       FROM localhost/bnpi-common:latest
<missing>     17 minutes ago  /bin/sh -c #(nop) ENTRYPOINT ["/usr/bin/sy...  453 MB      FROM localhost/bnpi-base:latest
<missing>     20 minutes ago  |8 MODULES=all PYTHONDONTWRITEBYTECODE=1 a...  43.7 MB     FROM docker.io/library/debian:bullseye-slim
<missing>     2 weeks ago     /bin/sh -c #(nop) ADD file:4a063d4e089ef10...  84 MB       
(reverse-i-search)`im': podman ^Cages | head
$ podman history bnpi-site-main | grep -v '0 B'
ID            CREATED         CREATED BY                                     SIZE        COMMENT
d9b3ab7ca9d2  13 minutes ago  |7 MODULES=main PYTHONDONTWRITEBYTECODE=1 ...  630 MB      FROM localhost/bnpi-common:latest
<missing>     17 minutes ago  /bin/sh -c #(nop) ENTRYPOINT ["/usr/bin/sy...  453 MB      FROM localhost/bnpi-base:latest
<missing>     20 minutes ago  |8 MODULES=all PYTHONDONTWRITEBYTECODE=1 a...  43.7 MB     FROM docker.io/library/debian:bullseye-slim
<missing>     2 weeks ago     /bin/sh -c #(nop) ADD file:4a063d4e089ef10...  84 MB       

Manually built at the moment:
https://github.com/srd424/birdnetpi-docker/pkgs/container/bnpi-server/112165916?tag=layered
https://github.com/srd424/birdnetpi-docker/pkgs/container/bnpi-site-main/112162066?tag=layered
Edit: oops, a few bugs in those images - cron doesn't get installed, for one. Will revisit when I've recovered from my weekend.
Edit 2: fixed - I think...

Next steps probably:

• split out the streamlit stuff (which I don't think is working anyway at the moment) into another container
• make the containers create default config files / databases etc if they are not found on startup - this should make it easier to get started with these images
• make the containers docker/podman compose-able (bit of a learning curve for me as I'm not familiar with it.)

[srd424]

OK, I merged the shared layer stuff into the main branch, and the main packages (tagged :amd64) are now built that way.

[srd424]

I've updated the mainline modularized packages to populate the 'state' directory if required, including creating an empty database, and generating a default config file (with the GeoIP stuff re-enabled to get an approximate lat/lon.) Hopefully this is getting to the point where it might actually be genuinely useful and relatively easy to use now. Current TODO list:

• split out the streamlit stuff (which I don't think is working anyway at the moment) into another container
• make the containers create default config files / databases etc if they are not found on startup - this should make it easier to get started with these images done
• make the containers docker/podman compose-able (bit of a learning curve for me as I'm not familiar with it.)
• option to operate icecast & RTSP recording in "reverse" mode, so one can push an RTSP stream from a recording station to icecast, and have birdnet record its snippets from icecast. This feels like a more natural way to operate when running as a container, and as a side effect we can probably eliminate the forced transcode to mp3 for the livestream, which uses a bit of CPU time (presumably only if the input doesn't match - e.g. my setup where the incoming stream is Opus.)
• update documentation

[srd424]

Have had a first shot at pulling the streamlit stuff out into a separate container:

https://github.com/srd424/birdnetpi-docker/pkgs/container/bnpi-site-streamlit

I fiddled with the main site container so you can pass in the streamlit host/port by setting environment variable BNPI_STREAMLIT - can be done with -e option on podman/docker command line, e.g. on my testing system:

podman run --replace --no-hosts --dns=192.168.128.2 --name birdnet -e BNPI_STREAMLIT=192.168.128.50:8501 -p 8088:80 -v $HOME/src/birdnet.state:/state -t -i bnpi-site-main:latest

I haven't really made any attempt to re-optimize the layers to take this into account yet, so there's probably some space that can be shared/saved between the different images, once I can face another round of staring in confusion at screens full of pip dependency trees :)

[srd424]

I've pushed a new version with an extra shared layer that saves ~200MB if you have all three images installed - total size is now about 1.2GB I think which doesn't seem too bad. Any further significant improvements will probably involve doing terrible things inserting custom layers with umoci so probably not worth it :)

[srd424]

I updated the README, hopefully now it consolidates everything I've said here and makes it reasonably obvious how to use the modularized images:
https://github.com/srd424/birdnetpi-docker/blob/main/README.md
If the README is still unclear to anybody please let me know!
I've also deleted the older 'monolithic' image as I won't be maintaining that.

[matthew73210]

Any chance of using docker-compose? For arm you just need to use agrs flags have have docker sort it out (can it still do that?)

[srd424]

For the build? Not familiar with that but I can have a look. I also think my docs are still a bit wrong, I think the current server image expects to run in "multi site" mode and but the docs are still for single site.

I'd sort of paused work because it was working well enough for my needs and there didn't seem to be much interest, but if an arm build would be useful I can definitely have a go at that at some point - one way or another it shouldn't be hard.

[matthew73210]

There's always birdcage, which isn't too dissimilar to birdpi. The basis is the same just docker and no extra fluff. I've contributed a little bit but on night shifts I don't have the energy to code right now 🫣

[srd424]

Ditto, the weather here has melted my few remaining brain cells, plus distracted by real-world things and other projects!

I'll definitely take a leaf out of your book and look at talking to a (slightly patched?) BirdNET-Analyzer instance for the analysis server, I'm not sure why birdnet went its own way on that, but if it's in maintenance mode it probably makes sense to track the original "upstream" for the analysis.