Snapcast: local deb installation competing for priority with bookworm-backports apt repository

[cbrenberg]

Creating a bug report/issue

• I have searched the existing open and closed issues

Required Information

• DietPi version | 9.9.0
• Distro version | bookworm
• Kernel version | 6.1.21-v8+
• SBC model | RPi3B+
• Power supply used | plenty of power, unrelated to issue
• SD card used | again, unrelated

Additional Information (if applicable)

• Software title | snapserver
• Was the software title installed freshly or updated/migrated? yes
• Can this issue be replicated on a fresh installation of DietPi? not sure

Steps to reproduce

1. via apt, upgrade snapserver package from 0.29.0 to 0.30.0
2. try to start the snapserver daemon via systemctl
3. snapserver fails to start due to a user/group name change causing directory permissions issues (details below)

Expected behaviour

• the snapserver daemon should have permissions to access its files within /var/lib/snapserver

Actual behaviour

• daemon crashes on startup due to lack of permissions to access its config directory at /var/lib/snapserver
• permissions on that directory under v0.29.0 and below were assigned to a snapserver user and group, whereas the systemctl unit for v0.30.0 uses a _snapserver user and group (note the leading underscore)

Extra details

• not sure if dietpi runs any post-install scripts for apt packages outside of dietpi version upgrades, but seems to me that either the systemctl unit should be modified to target the same user and group name as the previous version, OR ownership of /var/lib/snapserver should be changed to the new user and group name.

I didn't see anything in the snapserver release notes related to a user or group name changing, so not sure where in the whole pipeline this change took place. Sorry if it's not a dietpi thing.

[MichaIng]

Many thanks for your report.

Did you upgrade the package via APT from Bookworm backports repo? Because Bookworm should offer v0.26.0 only: https://packages.debian.org/snapserver

Background: not the Snapcast version is relevant, but whether it is the package from Debian, or the one from Snapcast themselves: https://github.com/badaix/snapcast/releases/
Debian uses _snapserver (don't ask me why ...), and Snapcast uses snapserver.

When (re)installing it via dietpi-software, we actively migrate the permissions of /var/lib/snapserver, according to which package is installed. But of course we cannot interfere when the Debian package is installed manually. This however should never happen, unless one does a Debian distro upgrade (from Bookworm to Trixie/testing), or fulls the package actively from backports.

[cbrenberg]

You're right! At some point I must have added the bookworm backports repo for reasons I cannot recall.

I vaguely remember having problems a long time ago getting the snapweb interface up and running after noticing the snapweb directory wasn't being included in the snapserver installation available from dietpi-software, so maybe I added the backports repo manually to see if that issue was fixed in the latest version of snapserver at that point in time.

In any case, I'll close this out as it doesn't seem to be related to dietpi. Thanks for pointing me in the right direction!

[MichaIng]

The backports repo is present OOTB, but by default, packages are not installed from there. It would require either

apt install -t bookworm-backports snapserver

or a change or the priority for this repository via /etc/apt/preferences.d config.

Also snapweb is missing in the Debian package only (we install it separetely in dietpi-software code on Trixie systems in the meantime), but included in the packages from Snapcast. So on Bookworm, it should have been always present. Or maybe your instance was installed before Bookworm was released, resp. before Snapcast provided Bookworm packages?

Ah yeah, found it now, before April last year, on Bookworm, the snapweb was missing: 75e113d
And then we recognised and added the missing snapweb for those cases: 2b34d77

Coincidentally, Snapcast 0.31.0 was just released an hour ago: https://github.com/badaix/snapcast/releases/tag/v0.31.0
So you could upgrade with a reinstall:

dietpi-software reinstall 191

It will adjust the /var/lib/snapserver again 😄.

[cbrenberg]

That's very odd. I definitely didn't target anything but the default repository in my apt command, just normal 'apt upgrade', nor have I manually changed repo priority. /etc/apt/preferences.d is entirely empty.

I can't say for certain, but it's possible this device's original dietpi installation may predate bookworm, so maybe my repo priority got messed up behind the scenes somewhere along the way and I'm only noticing it now. No idea.

I've had other recurring issues with mpd and mympd requiring manual intervention to get their systemd units to start up successfully after apt upgrades and dietpi version upgrades, which I suppose could also have a similar cause.

I'll try reinstalling via dietpi-software and hope that clears it up.

[MichaIng]

Can you show the output of this:

apt policy snapserver snapclient

It shows the priorities of the packages from each repository and the installed one. By default, the installed one and those from main, security and updates repos should be all 500, while the backports one should be 100. APT then picks the newest version from those sources with the highest priority, i.e. by default skips the one from backports.

myMPD is not available in the Debian repo, and we do not install MPD from other sources, so the issues there should not be related to backports.

[cbrenberg]

Here's the output after reinstalling both via dietpi-software. Somehow backports has equal priority to the local deb that dietpi-software downloaded from the github release.

snapserver:
  Installed: 0.31.0-1
  Candidate: 0.31.0-1
  Version table:
 *** 0.31.0-1 100
        100 /var/lib/dpkg/status
     0.30.0-1~bpo12+1 100
        100 https://deb.debian.org/debian bookworm-backports/main arm64 Packages
     0.26.0+dfsg1-1+deb12u1 500
        500 https://deb.debian.org/debian-security bookworm-security/main arm64 Packages
     0.26.0+dfsg1-1+b2 500
        500 https://deb.debian.org/debian bookworm/main arm64 Packages
snapclient:
  Installed: 0.31.0-1
  Candidate: 0.31.0-1
  Version table:
 *** 0.31.0-1 100
        100 /var/lib/dpkg/status
     0.30.0-1~bpo12+1 100
        100 https://deb.debian.org/debian bookworm-backports/main arm64 Packages
     0.26.0+dfsg1-1+deb12u1 500
        500 https://deb.debian.org/debian-security bookworm-security/main arm64 Packages
     0.26.0+dfsg1-1+b2 500
        500 https://deb.debian.org/debian bookworm/main arm64 Packages

[MichaIng]

Oh, I just checked back, and indeed the currently installed package always has priority 100. Then it makes all sense, as packages from backports are never picked automatically for a fresh install, but for upgrades: https://manpages.debian.org/bookworm/apt/apt_preferences.5.en.html

If the target release has not been specified then APT simply assigns priority 100 to all installed package versions and priority 500 to all uninstalled package versions, except versions coming from archives which in their Release files are marked as "NotAutomatic: yes" - these versions get the priority 1 or priority 100 if it is additionally marked as "ButAutomaticUpgrades: yes".

APT then applies the following rules, listed in order of precedence, to determine which version of a package to install.

• Never downgrade unless the priority of an available version exceeds 1000. ("Downgrading" is installing a less recent version of a package in place of a more recent version. Note that none of APT's default priorities exceeds 1000; such high priorities can only be set in the preferences file. Note also that downgrading a package can be risky.)
• Install the highest priority version.
• If two or more versions have the same priority, install the most recent one (that is, the one with the higher version number).
• If two or more versions have the same priority and version number but either the packages differ in some of their metadata or the --reinstall option is given, install the uninstalled one.

The question then is, why APT does not pull upgrades from backports right after the package has been installed from the regular suite. Once the package has been installed, it has priority 100, same as the one from backports, so both have same priority, but the one from backports has the higher version number 🤔.

... the reason is that in this case, the identical version from the regular suite would be no downgrade and still beats the backports in priority, even that it is identical to the installed version, hence no upgrade is done. But once a version is installed that is newer than the one from the regular suite, then it would be a downgrade, hence APT does not consider it anymore, as per the first rule above. And then, the remaining packages are the installed one and that from backports, both priority 100, hence backports wins.

Actually quite an important find: when we install DEB packages from other sources than APT repositories, we must keep in mind that they would be replaced by Debian backports, once it gets a newer version there. Usually this is not a problem, as config files and data are preserved, and the packages otherwise do not differ significantly. But here, due to the different user and group, it breaks things. I think we should place an /etc/apt/preferences.d config, to set backports priority for this particular package to 99, to assure it is never upgraded, or maybe even to -1, to assure it is never installed, unless it has been uninstalled via dietpi-software first (which will remove that config as well, of course).

[cbrenberg]

Agreed. Seems there's more nuance to these direct deb installations than has been previously considered. I agree that lowering the priority of Debian backports for this specific package seems like a reasonable solution to avoid the mismatched user/group naming issue.

But that also makes me wonder about package upgrades in this type of scenario where packages are installed from sources other than a proper apt repo. Let's say Snapcast publishes v0.32.0 next week.

If I have previously installed v0.31.0 via dietpi-software, there's no easy upgrade path to 0.32.0 for me without manually reinstalling and potentially reconfiguring via dietpi-software or waiting for the next version of dietpi to install the next version for me. Apt would also be unable to notify me that 0.32.0 is available, because none of the configured apt repositories contains the new version.

In most cases that's probably fine and ensures stability and compatibility with the rest of dietpi, but for users who prefer upgrading immediately to get access to the latest and greatest features, this installation pathway has barriers.

[MichaIng]

For a lot of software options, a reinstall is indeed the intended method to update it. But for Snapcast it is indeed missing in the docs. Will add it. This preserves configs etc.

DietPi updates do almost never update software versions (outside of those present in APT repos), unless some sort of migration is needed wo do not want to have in dietpi-software code, or it is security-relevant.