BirdnetPi vs iOS app

[ExperiMentor]

I have a Pi 02W running BirdnetPi and an iPad running the analagous Birdweather app. They have identical microphones located a couple of feet apart. But the iOS app is detecting around 20 times more birdsongs every day, consistently. It's the same if I switch the microphones over.
Different 'Settings' are available on the 2 systems, but I have adjusted them as similarly as I can see how (eg confidence %age).
I don't expect identical results, but this difference seems huge. What is wrong?
What are the suggested 'Settings' for BirdnetPi?

[Fish31520]

Hi,

3 months ago, @ehpersonal38 replied this, it might interest you :

"I do suspect that the low confidence behavior you're seeing is due to the fact that BirdNET-Pi is currently using a model that was last updated about 2 years ago whereas the BirdNET mobile app (and I'm sure Merlin too) gets regular model updates. Patrick has begun work as of last month on implementing the new model, though I suspect it won't be fully ready for another several months."

[mcguirepr89]

This is mostly correct -- BirdNET-Pi is the only BirdNET iteration of which I know that uses the TFLite model. It is deprecated in favor of the newer models, according to its author, Stefan Kahl (@kahst).

I have discontinued, however, the development for the newer model, so you should not ever expect that from me.

[jflyons]

Hi @ExperiMentor - I have nearly the same dual setup as you describe. I have over the past weeks been using the Birdweather app on my fairly old iPad, no external microphone, letting it run during daylight hours (which even in freezing temps goes a full day without a power recharge). It picks up many more detections than does my BirdNET-Pi, with a mic in an opposite side yard. Your more direct comparison is very interesting, as mine's a bit apples and oranges. Overflying geese, ducks, gulls and corvids can be detected on both, but not the land/garden birds. I received great advice particularly from @ehpersonal38 on adjusting the internal settings in BirdNET-Pi to species-specific (and lower than .7) levels. But like you, I wonder what else I can do to understand their respective performance.

Overall, I must add that BirdNET-Pi starting mid-2022 and Birdweather later, have been fantastic tools in helping me pursue a life-long quest of knowing my birds!

[nilspupils]

@ExperiMentor: Please have a look at this comment for clarification:
#69 (reply in thread)

[ehpersonal38]

If you'd like to test the new model please see this: #184 (comment)

[DD4WH]

@ExperiMentor
The "number of birdsongs detected" is surely not a good indicator of the quality of a classifier. Did you have a look how many of them are false positives ?
And it depends on the aim you would like to achieve with your system, e.g. how many false positives are acceptable for your aim?

Your statement "the TFlite model used in BirdNET-Pi has been deprecated as it performs much less well than the full TensorFlow model" is surely wrong, as is shown in this paper: https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.14003
Have a look at figure 1.

The testing of the quality of different classifiers is very difficult and data- and time-consuming. So I would be very careful about statements on one model being much less performant as another model.

[ExperiMentor]

I agree that classification of relative performance is difficult. My point was that the 2 systems, promoted as being similar, give radically different results. Going that far is easy.
The 'deprecated' quote is from the BirdNET-Pi GitHub - ie it's own developers are saying 'should no longer be used'.

[ExperiMentor]

I haven't attempted any formal performance analysis. But the 20-fold difference in frequency applies across the board, including the 'Top 5' which are appropriate and correct species near me (eg more than half on both systems are European Robin, which are prevalent here at this time of year). And yet one system reports 20x more of them than the other, even when same confidence selected.