Fail strategies: [<HWTool.IPMI: 'ipmi'>] on focal on Dell PowerEdge R650xs

[przemeklal]

On Dell PowerEdge R650xs running on focal:

unit-hardware-observer-0: 15:43:07 WARNING unit.hardware-observer/0.juju-log Strategy <hw_tools.IPMIStrategy object at 0x7f8268f5ad60> install fail: Could not install package(s) [['freeipmi-tools=1.6.9-2~bpo20.04.1']]: None
unit-hardware-observer-0: 15:43:07 INFO unit.hardware-observer/0.juju-log Fail strategies: [<HWTool.IPMI: 'ipmi'>]

Charm is blocked, all IPMI checks are failing.

Revision 22 of hardware-observer.

[przemeklal]

I ran apt install freeipmi-tools on affected machines manually without any issues, the charm stayed in the blocked state but at least IPMIDCMICommandFailed alerts got resolved.

[jneo8]

What is the package version you manually install?

[przemeklal]

@jneo8 1.6.4-3ubuntu1.1 gets installed.

Trying to manually install the same version as the charm results in:

$ sudo apt install freeipmi-tools=1.6.9-2~bpo20.04.1
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 freeipmi-tools : Depends: freeipmi-common (= 1.6.9-2~bpo20.04.1) but 1.6.4-3ubuntu1.1 is to be installed
E: Unable to correct problems, you have held broken packages.

[jneo8]

The 1.6.9-2~bpo20.04.1 which be chosen by the apt lib, seems not able to be installed on this machine and it's not the version which normal apt command choose. I suggest we review the behavior of apt lib and to see what can we do to let apt lib choose the same version as apt command.

(I think we are able to reproduce this issue with some old version focal image)

[przemeklal]

If it helps, the "bad" package comes from focal-backports:

$ sudo apt-cache policy freeipmi-tools
freeipmi-tools:
  Installed: 1.6.4-3ubuntu1.1
  Candidate: 1.6.4-3ubuntu1.1
  Version table:
     1.6.9-2~bpo20.04.1 100
        100 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages
 *** 1.6.4-3ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.6.4-3ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages