No module named 'pysolarmanv5' on containerized HA after update from 24.11.3 to 24.12.1

[Xnyle]

Description

Solarman (https://github.com/davidrapan/ha-solarman) can no longer be loaded due to "missing" requirement

ERROR (MainThread) [homeassistant.config_entries] Error occurred loading flow for integration solarman: No module named 'pysolarmanv5'

As I'm not sure on which side the problem lies (as I have no clue how python module loading is supposed to work) I also opened home-assistant/core#132558

Reproduction Steps

Try to load Integration on HA Core Docker (arm64)

Log

ERROR (MainThread) [homeassistant.config_entries] Error occurred loading flow for integration solarman: No module named 'pysolarmanv5'

Version

2024.12.1

[davidrapan]

One of these will help:

• Log into ssh of your HA and run ha core rebuild
• Upgrade of HA core
• If you already are on the most recent version then cycling downgrade and upgrade back again, will help.

[Xnyle]

@davidrapan did you read "HA Core Docker (arm64)"
There is no ssh and I'm pretty sure even if i try to run ha core rebuild inside container it wont work / help much.

2024.12.1 also is the last stable version distributed via docker image (ghcr.io/home-assistant/home-assistant:stable)

home-assistant/core#132558 is already closed, they state it's the components fault.

As far as i understand it, there's just a library missing that should be there but also no new core release available via docker img that fixes that?

[davidrapan]

That was very quick and blatant response from their part w/o even looking at it I guess. There is nothing I can do from the components part which could fix it. I'm just stating for the HA what python modules I need and it's HA's responsibility to load them for me (integration).

The only thing which could be wrong from integration POV is having requirement to nonexistent python module or it's version which is clearly not the case here.

Edit: You need to force HA to reevaluate python requirements somehow. 😖

Edit2:

There is no ssh

There is always SSH.

[Xnyle]

Edit: You need to force HA to reevaluate python requirements somehow. 😖

Edit2:

There is no ssh

There is always SSH.

It's a container. It runs python3 -m homeassistant, thats it.

ha core rebuild seems to do exactly that, rebuild the core container. Its doing that outside of the core container on a full distribution. And it's probably only working cause you end up with something newer than 24.12.1 like 24.12.6 or something.

Other than that I have no idea how to fix dependency problems. The only non ephemeral storage a docker installation has is its sensor / configuration database

[davidrapan]

So it's also w/o support for addons?

[Xnyle]

Yes, addons != components. custom components are still supported and solarman was working just fine before.

My problem is, that apparently everyone is just poking around in the dark. I've no clue how on a docker container without any persistent plugin storage space or anything pip modules are being initialized. It has to be done at every startup and that mechanism seems to just nor work properly anymore for certain modules.

[davidrapan]

Ah I see!

Did you just simply try to remove the integration, restart HA and then install back the most recent version? Latest version also involves upgrade of pysolarmanv5 to 3.0.5 so it could maybe trigger something within HA to fulfill python requirements.

[Xnyle]

Yes removed integration, redownloaded it and then just now i realized the storage also contains a deps folder where python packages are downloaded to.
So i deleted pvsolarman5 from there.

And yes, when you try to add the Solarman component again you end up with new folders there pysolarmanv5 pysolarmanv5-3.0.5.dist-info.

But the errors described in the first post persist, so back at square one.

[davidrapan]

What if you manually remove that >=3.0.5 part?

[davidrapan]

When you try through docker cli: pip show pysolarmanv5 what does it says?

[Xnyle]

Then is says it's not installed. And it isn't cause pip uses the default env and installs packages to /usr/local/lib/python3.13/site-packages

Whereas HA seems to install all packages it downloads at runtime in /config/deps (which is mounted)

If I then pip install pysolarmanv5 the package is ofc installed afterwards in /usr/local/lib/python3.13/site-packages

If then I restart the container lib is gone again (what else).

But interestingly pysolarmanv5 is the only lib of all custom components that I use that is not already there in the docker image (umodbus / pyyaml etc. are already part of the docker image). That's why it breaks at that point.

So my guess is they (HA) broke the mechanism of including downloaded python libs.

I looked further, the "/config/deps" folder was created a long time ago (2022-07-13) so that mechanism isn't something new.

So conclusion from what I have so far: They changed something regarding library paths with 2024.12, download additional libraries to /config/deps (probably as always) instead of the standard system path but then forget to include that path in the library path (or something like that).

Maybe you could be so kind and put that in other words and open a new ticket on their bugtracker, as I'm no python nor HA expert.

[davidrapan]

I'm honestly not really that familiar how HA handles it so your explanation in this case is as good as mine (if not better). 😉

Did you had the time to investigate this observed behavior with older version of HA in container?

[davidrapan]

refactor: Add submodule pysolarmanv5, I was thinking about this change for quite some time and because it will help in your case here it is.

Edit: It's in v24.12.08_7

[Xnyle]

No it worked with older versions. up to 23.11.x.
I was thinking, maybe it actually is the modules fault with the way you address said library. maybe HA requires you to "prefix" the include with some HA path or whatever. Sorry if thats bs, I'm no python expert, but maybe you could look into the HA docs regarding includes?

TY for the update though will try that tomorrow

[ajayramaswamy]

I had the same issue. I fixed it by ...

1. backup at 2024.11.3
2. update to 2024.12.1 -see errors, ha release notes speak of update to python 3.13
3. podman stop hass container
4. rm -fr /config/deps/* - there were a LOT of old modules for older python versions
5. add PYTHONPATH=/config/deps to container env
6. start again and it works OK so far.

[Xnyle]

That PYTHONPATH did the trick, where does that info come from? It's not mentioned in any HA doc known to me?

[davidrapan]

Cause it's not HA specific.

[Arnoudd]

Same issue here. I reverted to 2.12.06_4 and it works again.

[ajayramaswamy]

That PYTHONPATH did the trick, where does that info come from? It's not mentioned in any HA doc known to me?

home-assistant/core#127966 (comment)

and further down

https://community.home-assistant.io/t/wth-is-going-on-with-docker-container-installs-and-third-party-integrations-that-need-python-modules/806430

[ajayramaswamy]

home-assistant/core#132404 (comment)

More importantly does anyone know WHY this issue was introduced in 2024.10 in the first place?

The issue originates from the introduction of uv as the package manager we now use for Python.

[davidrapan]

The issue originates from the introduction of uv as the package manager we now use for Python.

Now we getting somewhere. I didn't even know there was a change like that... But the most insane part is that they closed many issues complaining about it as "custom component's fault" even when there was a change w/ package managing? Gross...

[mrboogiee]

good to see this issue having a root cause. How hard do you think it would be to resolve this @davidrapan? (also experiencing this issue... ;) )

I am currently fixing this by first wondering why my inverter is showing errors and then running pip install pysolarmanv5 every time my pod is restarted...

[davidrapan]

As I already said, the latest version does not have pysolarmanv5 requirement so it's "solved" already. 😉

[Xnyle]

That's not a solution but just another hack to mitigate another hack. ;-) There is a reason we use the concept of dependency management. And copy pasting is not dependency management.

Anyhow, root cause is that they totally f.cked up because of that other hack checking for (/.dockerenv) in order to check if they are running in a container.

In my case it's not docker, so things break. That's what happens when devs do stupid assumptions ;-)

So solutions is to mount an empty dir to /.dockerenv if you're not running on docker.

Thats ofc also another hack but less worse than setting PYTHONPATH (cause that breaks other stuff)

[Xnyle]

home-assistant/core#132404

Reasoning quite hilarious in places.

[mrboogiee]

So solutions is to mount an empty dir to /.dockerenv if you're not running on docker.

Just implemented this in my K8s environment as well and of course the question now becomes if that is the solution or the new version :D. There is nothing in the empty dir though... Might solve issues with other HACS things as well though so I'll leave it in for now ;)

[davidrapan]

I totally agree but in this integration's case it's not the only reason why I changed dependency to "source included". 😉

[Xnyle]

Closing as there is nothing more to do here.
Needs to be fixed in HA home-assistant/core#132404