Unavailable packages #4119

New Issue

Tatsh · 2024-11-25T07:22:50+01:00

Tatsh commented

2024-11-25 07:22:50 +01:00

These are no longer in the tree and only HA wants them:

   The following unavailable installed packages were found
             dev-python/colorlog-6.8.2
             dev-python/numexpr-2.9.0
             dev-python/orjson-3.10.7-r1
             dev-python/pydantic-core-2.25.0
             dev-python/zeroconf-0.135.0

These are no longer in the tree and only HA wants them: ``` The following unavailable installed packages were found dev-python/colorlog-6.8.2 dev-python/numexpr-2.9.0 dev-python/orjson-3.10.7-r1 dev-python/pydantic-core-2.25.0 dev-python/zeroconf-0.135.0 ```

Tatsh commented

2024-12-10 06:24:34 +01:00

Also these packages need to have their Rust dependency updated as virtual/rust is now masked.

dev-python/uv
dev-python/cryptography

Also I am not sure why uv has to be installed. I doubt this is necessary for HA to work.

Also these packages need to have their Rust dependency updated as `virtual/rust` is now masked. - `dev-python/uv` - `dev-python/cryptography` Also I am not sure why uv has to be installed. I doubt this is necessary for HA to work.

BillK commented

2024-12-18 01:04:55 +01:00

uv is essential with newer versions: the HA devs decided to use uv as the internal package installer for missing python deps ("/etc/homeassistant/deps"). While laudable for security, it is a very bumpy road (bugs, broken installations), especially as they are not interested in installs/images outside their official ones. :(

The error message some of the problems give indicates that passing "--system" allows the venv to be bypassed, but I have not been able to figure out how yet.
BillK

uv is essential with newer versions: the HA devs decided to use uv as the internal package installer for missing python deps ("/etc/homeassistant/deps"). While laudable for security, it is a very bumpy road (bugs, broken installations), especially as they are not interested in installs/images outside their official ones. :( The error message some of the problems give indicates that passing "--system" allows the venv to be bypassed, but I have not been able to figure out how yet. BillK

Tatsh commented

2024-12-18 02:08:35 +01:00

We have to convince HA to change their ways. It will be a huge effort though.

fariouche commented

2024-12-27 13:08:17 +01:00

Yeah, just upgraded my debug instance and see many errors because of the "uv" package manager (which I hope will be better than pip in the end)

Some of my modules are installed using homeassistant directly and not from this repo (I prefer having a core homeassistant with minimum modules, and install the needed modules myself depending on the server I install it)

Anyway, the system option is in my opinion not the correct solution (as it will install the packages in the system root filesystem and will require root permission)
Before uv, I believe HA was creating the venv using pip automatically. Most likely the old venv is incompatible with the uv's venv. I'm testing and let you know

Yeah, just upgraded my debug instance and see many errors because of the "uv" package manager (which I hope will be better than pip in the end) Some of my modules are installed using homeassistant directly and not from this repo (I prefer having a core homeassistant with minimum modules, and install the needed modules myself depending on the server I install it) Anyway, the system option is in my opinion not the correct solution (as it will install the packages in the system root filesystem and will require root permission) Before uv, I believe HA was creating the venv using pip automatically. Most likely the old venv is incompatible with the uv's venv. I'm testing and let you know

fariouche commented

2024-12-27 14:58:06 +01:00

passing the option "-e UV_SYSTEM_PYTHON=1" in the init script makes uv use the system option by default...
However, for some reason it cannot find certifi-3024.7.22 (that is installed in my system) so a lot of dependencies are failing now when starting HA

The deps folder is also populated with packages that are already in the system (like cryptography, exact same version as in the system)

passing the option "-e UV_SYSTEM_PYTHON=1" in the init script makes uv use the system option by default... However, for some reason it cannot find certifi-3024.7.22 (that is installed in my system) so a lot of dependencies are failing now when starting HA ![image](/attachments/5f8400ea-26ac-45fb-86b9-40d980bec642) The deps folder is also populated with packages that are already in the system (like cryptography, exact same version as in the system)

image.png

100 KiB

fariouche commented

2024-12-27 15:58:30 +01:00

certifi-3024.7.22 does not exist in pip's repo... the latest is 2024.12.14, and since uv is ignoring the system installed packages, it fails for me. So I think that finding a way to let uv take system packages into consideration should fix the issue.

Anyway, I was able to install the spotipy package manually with no issues using uv install --system --target ${HA_HOME} spotipy==2.23.0

certifi-3024.7.22 does not exist in pip's repo... the latest is 2024.12.14, and since uv is ignoring the system installed packages, it fails for me. So I think that finding a way to let uv take system packages into consideration should fix the issue. Anyway, I was able to install the spotipy package manually with no issues using uv install --system --target ${HA_HOME} spotipy==2.23.0

fariouche commented

2024-12-27 18:00:17 +01:00

sorry for all the messages... this is driving me crazy.
So, I discovered that the system option of uv pip is not intended to work with target option.
A venv is mandatory. And even doing so, the venv will ignore any system packages and will hapilly install them in the target directory or in the venv site-package.
I haven't found anything to ignore this behavior. It seems like all the purpose of this ebuild is gone as it will download almost all the deps (at least in my case) even if they are already installed

sorry for all the messages... this is driving me crazy. So, I discovered that the system option of uv pip is not intended to work with target option. A venv is mandatory. And even doing so, the venv will ignore any system packages and will hapilly install them in the target directory or in the venv site-package. I haven't found anything to ignore this behavior. It seems like all the purpose of this ebuild is gone as it will download almost all the deps (at least in my case) even if they are already installed

Tatsh commented

2024-12-27 19:02:46 +01:00

We have to patch Home Assistant then. There's no other way. I want it to not need uv or anything similar. We use Portage. Anything that makes Home Assistant download deps needs to be patched out.

fariouche commented

2024-12-27 23:23:31 +01:00

The thing I dont like with HA is that it does the job of a package manager, and it should not, it's not its job, but the distro.
they should, imo, split the core from the package manager and the os.

The thing I dont like with HA is that it does the job of a package manager, and it should not, it's not its job, but the distro. they should, imo, split the core from the package manager and the os.

Tatsh commented

2024-12-27 23:36:36 +01:00

The issue is they think it's better if they manage all the dependencies and packaging with Docker containers. I really dislike when projects act this way (this is basically the same as vendoring). They refuse to actually support a normal distro-based installation.

The ebuilds here unpack most of that and we just need to do a little more to match up. I am not too afraid of this getting too complex yet. Python is much easier to package in Portage vs something like Haskell or Ruby. The only issue is if we create our own tarballs for whatever reason we need a place to host them and it's extra maintenance.

The issue is they think it's better if they manage all the dependencies and packaging with Docker containers. I really dislike when projects act this way (this is basically the same as vendoring). They refuse to actually support a normal distro-based installation. The ebuilds here unpack most of that and we just need to do a little more to match up. I am not too afraid of this getting too complex yet. Python is much easier to package in Portage vs something like Haskell or Ruby. The only issue is if we create our own tarballs for whatever reason we need a place to host them and it's extra maintenance.

fariouche commented

2024-12-28 12:11:20 +01:00

to sumarize what i understood so far.
Since they use uv to replace pip, i see that system installed packages are ignored.
they use uv pip the same way as pip, so the regression is actually comming from uv.

uv pip is called with --target option. The first difference is that the deps folder is now equivalent to site-packages (no more lib, bin, share folders)
uv does not support user mode that was used preciously. So a venv is required except if we pass the --system option (or the UV_SYSTEM_PYTHON=1 env)
However, combined with the target option does not have the expect behavior. The combination just makes uv use the system pyrhon, but use the packages installed in target only ignoring the system packages.

It seems that the uv errors are not preventing HA from running, so if the system has all the packages, it may be fine to run it like this (more testing is needed, not sure for example if the log file will be filled with errors over time etc)
In that case, not using the system python option is maybe desirable are you are sure that no addional packages are installed

For people like me that relies on a generic rootfs and install integrations manually from HA depending on the server, i have no solution yet.

And even with uv installing all the duplicate packages, I still see error of integrations not loading like pyipp for printers.

Oh, almost forgot: creating a venv is not a solution too. When creating a venv with system package enable, only python will use it. uv will still ignore the system packages by design

to sumarize what i understood so far. Since they use uv to replace pip, i see that system installed packages are ignored. they use uv pip the same way as pip, so the regression is actually comming from uv. uv pip is called with --target option. The first difference is that the deps folder is now equivalent to site-packages (no more lib, bin, share folders) uv does not support user mode that was used preciously. So a venv is required except if we pass the --system option (or the UV_SYSTEM_PYTHON=1 env) However, combined with the target option does not have the expect behavior. The combination just makes uv use the system pyrhon, but use the packages installed in target only ignoring the system packages. It seems that the uv errors are not preventing HA from running, so if the system has all the packages, it may be fine to run it like this (more testing is needed, not sure for example if the log file will be filled with errors over time etc) In that case, not using the system python option is maybe desirable are you are sure that no addional packages are installed For people like me that relies on a generic rootfs and install integrations manually from HA depending on the server, i have no solution yet. And even with uv installing all the duplicate packages, I still see error of integrations not loading like pyipp for printers. Oh, almost forgot: creating a venv is not a solution too. When creating a venv with system package enable, only python will use it. uv will still ignore the system packages by design

Tatsh commented

2025-01-25 15:41:45 +01:00

Updating here:

   The following unavailable installed packages were found
             dev-python/anyio-4.6.0
             dev-python/bcrypt-4.2.0-r3
             dev-python/colorlog-6.8.2
             dev-python/deepdiff-7.0.1
             dev-python/numexpr-2.9.0
             dev-python/orjson-3.10.7-r1
             dev-python/psutil-6.0.0-r1
             dev-python/pydantic-core-2.25.0
             dev-python/pyudev-0.24.1
             dev-python/zeroconf-0.135.0

Updating here: ``` The following unavailable installed packages were found dev-python/anyio-4.6.0 dev-python/bcrypt-4.2.0-r3 dev-python/colorlog-6.8.2 dev-python/deepdiff-7.0.1 dev-python/numexpr-2.9.0 dev-python/orjson-3.10.7-r1 dev-python/psutil-6.0.0-r1 dev-python/pydantic-core-2.25.0 dev-python/pyudev-0.24.1 dev-python/zeroconf-0.135.0 ```

brulzki commented

2025-01-26 02:58:08 +01:00

I've taken the approach of patching the homeassistant-min-2024.10.4.ebuild to accept minimum dependency version rather than specific versions, eg

-       ~dev-python/anyio-4.6.0[${PYTHON_USEDEP}]
+       >=dev-python/anyio-4.6.0[${PYTHON_USEDEP}]

I guess that's ignoring the package constraints set by HomeAssistant, but in my opinion, depending on a specific version of every package without good reason (like a known breakage in a later dependency) is just poor form.

Anyhow, it has built and seems to be running without issues so far. Having said that, I am only using a very small subset of HA, so testing is not extensive.

I've taken the approach of patching the homeassistant-min-2024.10.4.ebuild to accept minimum dependency version rather than specific versions, eg ``` - ~dev-python/anyio-4.6.0[${PYTHON_USEDEP}] + >=dev-python/anyio-4.6.0[${PYTHON_USEDEP}] ``` I guess that's ignoring the package constraints set by HomeAssistant, but in my opinion, depending on a specific version of every package without good reason (like a known breakage in a later dependency) is just poor form. Anyhow, it has built and seems to be running without issues so far. Having said that, I am only using a very small subset of HA, so testing is not extensive.

onkelbeh commented

2025-02-18 13:20:59 +01:00

12b1e4c20 dev-python/pyudev: add 0.24.1
5c8ea5def dev-python/pydantic-core: new package, add 2.25.0-r1
d684305a2 dev-python/psutil: add 6.0.0-r1 from main repo's archive
4b22960f5 dev-python/orjson: add 3.10.7-r1 (from main repo copy)
bbb855ffc dev-python/deepdiff: add 7.0.1
3323e1931 dev-python/bcrypt: drop 4.0.1, add 4.2.0-r3 (from main repo copy)
5e9b895e8 dev-python/anyio: add 4.6.0
0d223d067 dev-python/colorlog: add 6.8.2
caa838b08 dev-python/numexpr: new package, add 2.9.0
de6b59b98 dev-python-pydantic: add 1.10.18

12b1e4c20 dev-python/pyudev: add 0.24.1 5c8ea5def dev-python/pydantic-core: new package, add 2.25.0-r1 d684305a2 dev-python/psutil: add 6.0.0-r1 from main repo's archive 4b22960f5 dev-python/orjson: add 3.10.7-r1 (from main repo copy) bbb855ffc dev-python/deepdiff: add 7.0.1 3323e1931 dev-python/bcrypt: drop 4.0.1, add 4.2.0-r3 (from main repo copy) 5e9b895e8 dev-python/anyio: add 4.6.0 0d223d067 dev-python/colorlog: add 6.8.2 caa838b08 dev-python/numexpr: new package, add 2.9.0 de6b59b98 dev-python-pydantic: add 1.10.18

onkelbeh self-assigned this 2025-02-18 13:23:11 +01:00

Sign in to join this conversation.

5 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: onkelbeh/HomeAssistantRepository#4119