File collisions: dev-python/whois and dev-python/python-whois (different packages, both from here) #258

Closed
opened 2022-02-03 07:05:06 +01:00 by Tatsh · 24 comments

I attempted to upgrade to the latest homeassistant-min which, with my USE flags, requires dev-python/whois. dev-python/python-whois is already installed on my machine. I am not sure of the USE flags pulling both in.

I would prefer to not have to pick things based on potential dependencies. If these both have to be installed, then one should be renamed and the relevant code patched.

My USE flags:

accuweather
airly
airvisual
alpha_vantage
android_ip_webcam
androidtv
bluetooth_le_tracker
cast
coronavirus
github
homekit_controller
http
kodi
myq
-plex
ring
speedtestdotnet
systemmonitor
tile
tplink
wemo
whois

Log:

* This package will overwrite one or more files that may belong to other
 * packages (see list below).
 *
 * Detected file collision(s):
 *
 *      /usr/lib/python3.9/site-packages/whois/__init__.py
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.pyc
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-1.pyc
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-2.pyc
 *
 * Searching all installed packages for file collisions...
 *
 * Press Ctrl-C to Stop
 *
 * dev-python/python-whois-0.7.3:0::HomeAssistantRepository
 *      /usr/lib/python3.9/site-packages/whois/__init__.py
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-1.pyc
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-2.pyc
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.pyc
 *
 * Package 'dev-python/whois-0.9.13' NOT merged due to file collisions.
 * If necessary, refer to your elog messages for the whole content of the
 * above message.
I attempted to upgrade to the latest `homeassistant-min` which, with my USE flags, requires `dev-python/whois`. `dev-python/python-whois` is already installed on my machine. I am not sure of the USE flags pulling both in. I would prefer to not have to pick things based on potential dependencies. If these both have to be installed, then one should be renamed and the relevant code patched. My USE flags: ``` accuweather airly airvisual alpha_vantage android_ip_webcam androidtv bluetooth_le_tracker cast coronavirus github homekit_controller http kodi myq -plex ring speedtestdotnet systemmonitor tile tplink wemo whois ``` Log: ``` * This package will overwrite one or more files that may belong to other * packages (see list below). * * Detected file collision(s): * * /usr/lib/python3.9/site-packages/whois/__init__.py * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.pyc * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-1.pyc * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-2.pyc * * Searching all installed packages for file collisions... * * Press Ctrl-C to Stop * * dev-python/python-whois-0.7.3:0::HomeAssistantRepository * /usr/lib/python3.9/site-packages/whois/__init__.py * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-1.pyc * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-2.pyc * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.pyc * * Package 'dev-python/whois-0.9.13' NOT merged due to file collisions. * If necessary, refer to your elog messages for the whole content of the * above message. ```
onkelbeh added the
Cleanup
label 2022-02-05 10:50:50 +01:00
onkelbeh self-assigned this 2022-02-05 10:50:56 +01:00
Owner

Yes, this was changed upstream, current version is ~dev-python/whois-0.9.13 (05defe259d)
In fact it's the same package with a new name. I cannot remove ~dev-python/python-whois-0.7.3 now, because it is still referenced by old versions:

homeassistant-2021.10.7.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] )
homeassistant-2021.11.5.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] )
homeassistant-2021.12.10.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] )
homeassistant-2022.2.0.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] )
homeassistant-2022.2.1.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] )
homeassistant-2022.2.2.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] )

You could (temporarily) remove homeassistant from your world file to remove the old, and install the new build.

I would leave this ticket open as a reminder to remove it.

Yes, this was changed upstream, current version is ~dev-python/whois-0.9.13 (https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/05defe259da6593f3feb4c67b0fafd1f9b4efe3c) In fact it's the same package with a new name. I cannot remove ~dev-python/python-whois-0.7.3 now, because it is still referenced by old versions: ``` homeassistant-2021.10.7.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] ) homeassistant-2021.11.5.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] ) homeassistant-2021.12.10.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] ) homeassistant-2022.2.0.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] ) homeassistant-2022.2.1.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] ) homeassistant-2022.2.2.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] ) ``` You could (temporarily) remove homeassistant from your world file to remove the old, and install the new build. I would leave this ticket open as a reminder to remove it.
Owner

Will try to change the Ebuild to remove the old package during the Update. I have seen such operations during world updates often, but currently I do not explicitly know to trigger this behavior. I will try to forbid the old package in the new Ebuild, perhaps this helps.

Will try to change the Ebuild to remove the old package during the Update. I have seen such operations during world updates often, but currently I do not explicitly know to trigger this behavior. I will try to forbid the old package in the new Ebuild, perhaps this helps.
Author

I also would like to know how that works for my own overlay. That would be the solution to the issue.

I also would like to know how that works for my own overlay. That would be the solution to the issue.
Author

I updated and I am getting errors on startup regarding hass_frontend missing.

Feb 05 05:24:02 tatsh hass[3714322]: 2022-02-05 05:24:02 ERROR (MainThread) [homeassistant.setup] Error during setup of component frontend
Feb 05 05:24:02 tatsh hass[3714322]: Traceback (most recent call last):
Feb 05 05:24:02 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/setup.py", line 227, in _async_setup_component
Feb 05 05:24:02 tatsh hass[3714322]:     result = await task
Feb 05 05:24:02 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 339, in async_setup
Feb 05 05:24:02 tatsh hass[3714322]:     root_path = _frontend_root(repo_path)
Feb 05 05:24:02 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 314, in _frontend_root
Feb 05 05:24:02 tatsh hass[3714322]:     import hass_frontend
Feb 05 05:24:02 tatsh hass[3714322]: ModuleNotFoundError: No module named 'hass_frontend'
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of map. Setup failed for dependencies: frontend
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for map: (DependencyError(...), 'Could not setup dependencies: frontend')
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of my. Setup failed for dependencies: frontend
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for my: (DependencyError(...), 'Could not setup dependencies: frontend')
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of logbook. Setup failed for dependencies: frontend
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for logbook: (DependencyError(...), 'Could not setup dependencies: frontend')
Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 WARNING (MainThread) [homeassistant.bootstrap] Detected that frontend did not load. Activating safe mode
Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Error during setup of component frontend
Feb 05 05:24:06 tatsh hass[3714322]: Traceback (most recent call last):
Feb 05 05:24:06 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/setup.py", line 227, in _async_setup_component
Feb 05 05:24:06 tatsh hass[3714322]:     result = await task
Feb 05 05:24:06 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 339, in async_setup
Feb 05 05:24:06 tatsh hass[3714322]:     root_path = _frontend_root(repo_path)
Feb 05 05:24:06 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 314, in _frontend_root
Feb 05 05:24:06 tatsh hass[3714322]:     import hass_frontend
Feb 05 05:24:06 tatsh hass[3714322]: ModuleNotFoundError: No module named 'hass_frontend'
Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of safe_mode. Setup failed for dependencies: frontend
Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Setup failed for safe_mode: (DependencyError(...), 'Could not setup dependencies: frontend')
I updated and I am getting errors on startup regarding `hass_frontend` missing. ``` Feb 05 05:24:02 tatsh hass[3714322]: 2022-02-05 05:24:02 ERROR (MainThread) [homeassistant.setup] Error during setup of component frontend Feb 05 05:24:02 tatsh hass[3714322]: Traceback (most recent call last): Feb 05 05:24:02 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/setup.py", line 227, in _async_setup_component Feb 05 05:24:02 tatsh hass[3714322]: result = await task Feb 05 05:24:02 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 339, in async_setup Feb 05 05:24:02 tatsh hass[3714322]: root_path = _frontend_root(repo_path) Feb 05 05:24:02 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 314, in _frontend_root Feb 05 05:24:02 tatsh hass[3714322]: import hass_frontend Feb 05 05:24:02 tatsh hass[3714322]: ModuleNotFoundError: No module named 'hass_frontend' Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of map. Setup failed for dependencies: frontend Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for map: (DependencyError(...), 'Could not setup dependencies: frontend') Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of my. Setup failed for dependencies: frontend Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for my: (DependencyError(...), 'Could not setup dependencies: frontend') Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of logbook. Setup failed for dependencies: frontend Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for logbook: (DependencyError(...), 'Could not setup dependencies: frontend') Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 WARNING (MainThread) [homeassistant.bootstrap] Detected that frontend did not load. Activating safe mode Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Error during setup of component frontend Feb 05 05:24:06 tatsh hass[3714322]: Traceback (most recent call last): Feb 05 05:24:06 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/setup.py", line 227, in _async_setup_component Feb 05 05:24:06 tatsh hass[3714322]: result = await task Feb 05 05:24:06 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 339, in async_setup Feb 05 05:24:06 tatsh hass[3714322]: root_path = _frontend_root(repo_path) Feb 05 05:24:06 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 314, in _frontend_root Feb 05 05:24:06 tatsh hass[3714322]: import hass_frontend Feb 05 05:24:06 tatsh hass[3714322]: ModuleNotFoundError: No module named 'hass_frontend' Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of safe_mode. Setup failed for dependencies: frontend Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Setup failed for safe_mode: (DependencyError(...), 'Could not setup dependencies: frontend') ```
Owner

This is my upgrade on hasstest from 2021.12.10 -> 2022.2.2,
it's the -min Ebuild with all useflags execpt systemd, ozw & tradfri turned on:

grafik

Soft-blocking in both directions seems to behave as expected:

Total: 41 packages (40 upgrades, 1 new, 1 uninstall), Size of downloads: 0 KiB
Conflict: 2 blocks (all satisfied)

Added this now in 1e9ce544cd

The 'new' whois avoids external libraries, which I appreciate, we have plenty of them, already ;-)

BTW: let me add that adopting homeassistant-2022.2.x in production is a bit early now, would give them a few more days, this is a very fat update, and they are currently receiving >150 tickets/day, 4 days in a row. My production box still is on 2021.12.10.

This is my upgrade on hasstest from `2021.12.10` -> `2022.2.2`, it's the -min Ebuild with all useflags execpt `systemd`, `ozw` & `tradfri` turned on: ![grafik](/attachments/5aad821f-fc94-4617-a1dc-f291fb8a6f86) Soft-blocking in both directions seems to behave as expected: ``` Total: 41 packages (40 upgrades, 1 new, 1 uninstall), Size of downloads: 0 KiB Conflict: 2 blocks (all satisfied) ``` Added this now in https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/1e9ce544cda07fee9598c9181fead406cc206fce The 'new' whois avoids external libraries, which I appreciate, we have plenty of them, already ;-) BTW: let me add that adopting homeassistant-2022.2.x in production is a bit early now, would give them a few more days, this is a very fat update, and they are currently receiving >150 tickets/day, 4 days in a row. My production box still is on 2021.12.10.
3.4 MiB
Author

I'll just downgrade back to 12.10 for now.

I'll just downgrade back to 12.10 for now.
Author

Actually I cannot downgrade now because requests-2.26.0 is now out of the main tree.

emerge: there are no ebuilds to satisfy "~dev-python/requests-2.26.0[python_targets_python3_8(-)?,python_targets_python3_9(-)?,python_targets_python3_10(-)?]".
(dependency required by "app-misc/homeassistant-min-2021.12.10::HomeAssistantRepository" [ebuild])
Actually I cannot downgrade now because requests-2.26.0 is now out of the main tree. ``` emerge: there are no ebuilds to satisfy "~dev-python/requests-2.26.0[python_targets_python3_8(-)?,python_targets_python3_9(-)?,python_targets_python3_10(-)?]". (dependency required by "app-misc/homeassistant-min-2021.12.10::HomeAssistantRepository" [ebuild]) ```
Owner

Yeah, confirm, this must be brand-new, 20 minutes ago this still worked. will re-add it, takes a few minutes, currently updating the git host.

Yeah, confirm, this must be brand-new, 20 minutes ago this still worked. will re-add it, takes a few minutes, currently updating the git host.
Owner

re-added in 560c630856

re-added in https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/560c630856a3234ec526f7ff3960c252b2deaf95
Author

Missing a file:

>>> Preparing source in /var/tmp/portage/dev-python/requests-2.26.0/work/requests-2.26.0 ...
 * Applying requests-2.26.0-test.patch ...
/var/tmp/portage/dev-python/requests-2.26.0/temp/environment: line 1652: /var/tmp/portage/dev-python/requests-2.26.0/files/requests-2.26.0-test.patch: No such file or directory
/var/tmp/portage/dev-python/requests-2.26.0/temp/environment: line 1655: /var/tmp/portage/dev-python/requests-2.26.0/files/requests-2.26.0-test.patch: No such file or directory
Missing a file: ``` >>> Preparing source in /var/tmp/portage/dev-python/requests-2.26.0/work/requests-2.26.0 ... * Applying requests-2.26.0-test.patch ... /var/tmp/portage/dev-python/requests-2.26.0/temp/environment: line 1652: /var/tmp/portage/dev-python/requests-2.26.0/files/requests-2.26.0-test.patch: No such file or directory /var/tmp/portage/dev-python/requests-2.26.0/temp/environment: line 1655: /var/tmp/portage/dev-python/requests-2.26.0/files/requests-2.26.0-test.patch: No such file or directory ```
Owner

Regarding the frontend issue:
in the current version of home-assistant-frontend-20220203.0, they decided to remove the SDIST on Pypi, which we used to use.

12c12,18
< SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz"
---
>
> # SDIST missing on Pypi
> #SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz"
> # use Github instead
> MY_PN="frontend"
> SRC_URI="https://github.com/home-assistant/${MY_PN}/archive/refs/tags/${PV}.tar.gz -> ${P}.tar.gz"
> S="${WORKDIR}/${MY_PN}-${PV}"

Seems there are problems with setuptools and the source from Github.
There is an open ticket at frontend: https://github.com/home-assistant/frontend/issues/11542

Regarding the frontend issue: in the current version of home-assistant-frontend-20220203.0, they decided to remove the [SDIST on Pypi](https://pypi.org/project/home-assistant-frontend/#files), which we used to use. ```diff 12c12,18 < SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz" --- > > # SDIST missing on Pypi > #SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz" > # use Github instead > MY_PN="frontend" > SRC_URI="https://github.com/home-assistant/${MY_PN}/archive/refs/tags/${PV}.tar.gz -> ${P}.tar.gz" > S="${WORKDIR}/${MY_PN}-${PV}" ``` Seems there are problems with setuptools and the source from Github. There is an open ticket at frontend: https://github.com/home-assistant/frontend/issues/11542
Owner

We should split this in two tickets ;-)

looked easy, only the Ebuild was removed in 676df577ce (diff-f721f8955904a02e32c432f4438fc7dea01bd8ecef896d7e35a58ec1a8b0a12d), did not look at the patches, and skipped the test, bad, resulted in a 46 packages to downgrade.
now it runs again, I copied the missing patches in 06106473f8

We should split this in two tickets ;-) looked easy, only the Ebuild was removed in https://github.com/gentoo/gentoo/commit/676df577ce9ac07480f7d6be702d23367161859b#diff-f721f8955904a02e32c432f4438fc7dea01bd8ecef896d7e35a58ec1a8b0a12d, did not look at the patches, and skipped the test, bad, resulted in a 46 packages to downgrade. now it runs again, I copied the missing patches in https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/06106473f8ccfdd1375f1675b72acd7d15760211
Author

Let me know when you want me to try the latest stable version of homeassistant-min.

Let me know when you want me to try the latest stable version of homeassistant-min.
Owner

Sure, I just removed the 2022.2.2 release, we will have to fix the currently broken frontend at first.

Sure, I just removed the 2022.2.2 release, we will have to fix the currently broken frontend at first.
Owner

Unfortunately I have only low to no experience how to include a yarn build into Ebuilds.

Unfortunately I have only low to no experience how to include a yarn build into Ebuilds.
Author

My suggestion is not to try that. Your scripts that handle generating the ebuilds could also run the build process of the UI and package that up for distribution, and put it somewhere you can host. This would be almost the same as the releases that had SDIST.

To really make Yarn work inside of Portage the normal way would require figuring out every single Yarn package URL that needs to be downloaded by Portage (not via Yarn) and listing them in SRC_URI. Then you have to tell Yarn to pull all packages in locally from DISTDIR. Then you can run the project's build command (presumably yarn build).

This is really not ideal at all because there will be thousands of tiny tgz files downloaded just for HA frontend. And this would be, for most people, the only package that has distfiles from Yarn. It just seems pointless.

I have done a similar thing with a package that uses Nuget and we see where this is going with Go and Rust, where every ebuild just has to pull in all deps. It is not so bad for those because usually it's less than 100 files to download. We have generators/eclasses for these things that work pretty well. And it's unlikely to have 50 different versions of the same package in DISTDIR from Rust and Go, or even Nuget.

Nuget example - I am giving this example because there is not yet a Nuget eclass to help, just like there is not yet something for NPM or Yarn.

In the case of NPM/Yarn, they have given up on trying to resolve dependency trees where each package is unique in the entire tree. You can have a package that relies on some dependency that pulls in a different version of another dependency. Imagine if in Python: (x requires requests-2.26)+(y requires requests-2.27) but instead of making the dev of x resolve this issue by using a similar constraint (requests>=2.27), instead they 'silo' the y package to use its own copy of requests-2.27. It's like 'vendoring' (as it's sometimes called in Go) but on a massive scale.

Alternatively, you can use pkg_setup() (or another non-sandboxed phase) to run yarn inside S. This will download all the dependencies (but without more arguments it will not respect proxies which might be set in FETCHCOMMAND). The files would of course be cleaned out once the installation is done. I don't think Gentoo would ever accept an ebuild that does this, but it will work.

My suggestion is not to try that. Your scripts that handle generating the ebuilds could also run the build process of the UI and package that up for distribution, and put it somewhere you can host. This would be almost the same as the releases that had SDIST. To really make Yarn work inside of Portage the normal way would require figuring out every single Yarn package URL that needs to be downloaded by Portage (not via Yarn) and listing them in `SRC_URI`. Then you have to tell Yarn to pull all packages in locally from `DISTDIR`. Then you can run the project's build command (presumably `yarn build`). This is really not ideal at all because there will be *thousands* of tiny tgz files downloaded just for HA frontend. And this would be, for most people, the only package that has distfiles from Yarn. It just seems pointless. I have done a similar thing with a package that uses Nuget and we see where this is going with Go and Rust, where every ebuild just has to pull in all deps. It is not so bad for those because usually it's less than 100 files to download. We have generators/eclasses for these things that work pretty well. And it's unlikely to have 50 different versions of the same package in DISTDIR from Rust and Go, or even Nuget. [Nuget example](https://github.com/Tatsh/tatsh-overlay/blob/d618e94cfa90cb74cb55ab7154671324c492785f/games-emulation/ryujinx/ryujinx-1.1.13.ebuild) - I am giving this example because there is not yet a Nuget eclass to help, just like there is not yet something for NPM or Yarn. In the case of NPM/Yarn, they have given up on trying to resolve dependency trees where each package is unique in the entire tree. You can have a package that relies on some dependency that pulls in a different version of another dependency. Imagine if in Python: (x requires requests-2.26)+(y requires requests-2.27) but instead of making the dev of x resolve this issue by using a similar constraint (`requests>=2.27`), instead they 'silo' the y package to use its own copy of requests-2.27. It's like 'vendoring' (as it's sometimes called in Go) but on a massive scale. Alternatively, you can use `pkg_setup()` (or another non-sandboxed phase) to run `yarn` inside `S`. This will download all the dependencies (but without more arguments it will not respect proxies which might be set in `FETCHCOMMAND`). The files would of course be cleaned out once the installation is done. I don't think Gentoo would ever accept an ebuild that does this, but it will work.
Owner

Yewah, thanks for the thoughts. We all know that npm does not mix well with portage. During compilation of the stuff for the git server updates, I was playing around, looks like I underestimated the volume of 'home-assistant-frontend', my biggest machine spends over 25 minutes compiling it from source, next we would need yarn & node-js installed. But it seems that I can get one working.

Now I have the part downloading in src_prepare(), it does not run in pkg_setup().

I want to know that we COULD do it, just in case.
Also I can checkout a way in downloading 'data' without having it in Manifest, we could use it in Ebuilds like dev-python/mac-vendor-lookup, where I currently have a big blob of a mac address table in the files directory.

There is a conversation going on with one of the frontend guys, looks like they are willing to change their build process and put an archive back online in the stage after the yarn build. Caused by an "out of space" issue, they can't put it back on Pypi, but I would not strain taking it from Github :-)

If they really put it back online, best way would be to let the user decide, switched by a USE flag.

On the other hand, messing around with npm in an Ebuild would also open the frontiers a bit more for Ebuilds like

  • zwavejs2mqtt
  • zigbee2mqtt
  • node-red

This is the experimental version:

## Copyright 1999-2022 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=8

PYTHON_COMPAT=( python3_{8..10} )

inherit distutils-r1
RESTRICT=network-sandbox

DESCRIPTION="The Home Assistant frontend"
HOMEPAGE="https://github.com/home-assistant/frontend https://pypi.org/project/home-assistant-frontend/"

# SDIST missing on Pypi
#SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz"
# use Github instead
MY_PN="frontend"
SRC_URI="https://github.com/home-assistant/${MY_PN}/archive/refs/tags/${PV}.tar.gz -> ${P}.tar.gz"
S="${WORKDIR}/${MY_PN}-${PV}"

LICENSE="Apache-2.0"
SLOT="0"
KEYWORDS="~amd64 ~arm ~arm64 ~x86 ~amd64-linux ~x86-linux"
IUSE="test"

RDEPEND="~dev-python/user-agents-2.0[${PYTHON_USEDEP}]"
BDEPEND="${REDEPEND}
	dev-python/setuptools[${PYTHON_USEDEP}]
	sys-apps/yarn
	test? (
		dev-python/nose[${PYTHON_USEDEP}]
		dev-python/pytest[${PYTHON_USEDEP}]
	)"

# Githubs source is lacking the built node modules
src_prepare() {
	yarn install || die
	eapply_user
}

src_compile() {
	./script/build_frontend || die
	default
}
Yewah, thanks for the thoughts. We all know that npm does not mix well with portage. During compilation of the stuff for the git server updates, I was playing around, looks like I underestimated the volume of 'home-assistant-frontend', my biggest machine spends over 25 minutes compiling it from source, next we would need yarn & node-js installed. But it seems that I can get one working. Now I have the part downloading in `src_prepare()`, it does not run in `pkg_setup()`. I want to know that we COULD do it, just in case. Also I can checkout a way in downloading 'data' without having it in Manifest, we could use it in Ebuilds like `dev-python/mac-vendor-lookup`, where I currently have a big blob of a mac address table in the files directory. There is a conversation going on with one of the frontend guys, looks like they are willing to change their build process and put an archive back online in the stage after the yarn build. Caused by an "out of space" issue, they can't put it back on Pypi, but I would not strain taking it from Github :-) If they really put it back online, best way would be to let the user decide, switched by a USE flag. On the other hand, messing around with npm in an Ebuild would also open the frontiers a bit more for Ebuilds like * zwavejs2mqtt * zigbee2mqtt * node-red This is the experimental version: ```bash ## Copyright 1999-2022 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 EAPI=8 PYTHON_COMPAT=( python3_{8..10} ) inherit distutils-r1 RESTRICT=network-sandbox DESCRIPTION="The Home Assistant frontend" HOMEPAGE="https://github.com/home-assistant/frontend https://pypi.org/project/home-assistant-frontend/" # SDIST missing on Pypi #SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz" # use Github instead MY_PN="frontend" SRC_URI="https://github.com/home-assistant/${MY_PN}/archive/refs/tags/${PV}.tar.gz -> ${P}.tar.gz" S="${WORKDIR}/${MY_PN}-${PV}" LICENSE="Apache-2.0" SLOT="0" KEYWORDS="~amd64 ~arm ~arm64 ~x86 ~amd64-linux ~x86-linux" IUSE="test" RDEPEND="~dev-python/user-agents-2.0[${PYTHON_USEDEP}]" BDEPEND="${REDEPEND} dev-python/setuptools[${PYTHON_USEDEP}] sys-apps/yarn test? ( dev-python/nose[${PYTHON_USEDEP}] dev-python/pytest[${PYTHON_USEDEP}] )" # Githubs source is lacking the built node modules src_prepare() { yarn install || die eapply_user } src_compile() { ./script/build_frontend || die default } ```
Owner

Mark Mueller @cdce8p just filed a PR, which should provide an SDIST for the frontend from the next release.

https://github.com/home-assistant/frontend/pull/11566

For now, I'll keep the home-assistant-frontend-20220203.0-r1.ebuild as a 'dirty hack' only for this release. It has networking allowed in the sandbox, and eats up 30 minutes build time.

Mark Mueller @cdce8p just filed a PR, which should provide an SDIST for the frontend from the next release. https://github.com/home-assistant/frontend/pull/11566 For now, I'll keep the home-assistant-frontend-20220203.0-r1.ebuild as a 'dirty hack' only for this release. It has networking allowed in the sandbox, and eats up 30 minutes build time.
Owner

HA's frontend team is currently refusing to provide the SDIST again. Perhaps: spread some likes:
https://github.com/home-assistant/frontend/issues/11542
https://github.com/home-assistant/frontend/pull/11566

HA's frontend team is currently refusing to provide the SDIST again. Perhaps: spread some likes: https://github.com/home-assistant/frontend/issues/11542 https://github.com/home-assistant/frontend/pull/11566
Owner
and https://github.com/onkelbeh/HomeAssistantRepository/issues/47
Author

I would like to update. Is a workaround in place? If not, I will continue waiting.

I would like to update. Is a workaround in place? If not, I will continue waiting.
Owner

as you suggested, I'll provide an own build of the frontend from now on, could you give it a try?

as you suggested, I'll provide an [own build](https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/ce2d37ebe89f0d5e92430da963d0f79193481946) of the frontend from now on, could you give it a try?
Owner

Seems to do it, 32 downloads per wget, no new tickets.

Seems to do it, 32 downloads per wget, no new tickets.
Owner

In the meantime ~50 downloads with no new tickets.
Hope this will stay like that with the next release.
I've upgraded my pruduction system to 2022.2.6 yesterday night as well.
Let' summarize this in #259, I'll close this one.

In the meantime ~50 downloads with no new tickets. Hope this will stay like that with the next release. I've upgraded my pruduction system to 2022.2.6 yesterday night as well. Let' summarize this in #259, I'll close this one.
Sign in to join this conversation.
No Label
Bug
Bump/Update
Cleanup
Dupe
Enhancement
File Collision
Forked
help wanted
Integration: accuweather
Integration: acmeda
Integration: acomax
Integration: adax
Integration: adguard
Integration: aemet
Integration: aep_ohio
Integration: aep_texas
Integration: airq
Integration: airthings
Integration: airthings_ble
Integration: airtouch5
Integration: airzone
Integration: airzone_cloud
Integration: amazon_polly
Integration: amberelectric
Integration: ambient_station
Integration: analytics_insights
Integration: anel_pwrctrl
Integration: aosmith
Integration: apache_kafka
Integration: apcupsd
Integration: appalachianpower
Integration: apprise
Integration: aprilaire
Integration: asuswrt
Integration: august
Integration: aws
Integration: axis
Integration: backup
Integration: bang_olufsen
Integration: blebox
Integration: blink
Integration: bluetooth
Integration: blue_current
Integration: bmw_connected_drive
Integration: bosch_shc
Integration: bring
Integration: brother
Integration: brottsplatskartan
Integration: bsblan
Integration: bthome
Integration: caldav
Integration: cast
Integration: ccm15
Integration: cloud
Integration: cloudflare
Integration: co2signal
Integration: coautilities
Integration: comelit
Integration: conversation
Integration: debugpy
Integration: deconz
Integration: deluge
Integration: denonavr
Integration: devialet
Integration: devolo_home_control
Integration: dhcp
Integration: discord
Integration: discovergy
Integration: dlna_dmr
Integration: dlna_dms
Integration: doods
Integration: drop_connect
Integration: dsmr
Integration: duotecno
Integration: duquesne_light
Integration: dwd_weather_warnings
Integration: easyenergy
Integration: ecobee
Integration: ecoforest
Integration: ecovacs
Integration: ecowitt
Integration: elgato
Integration: elvia
Integration: energyzero
Integration: enigma2
Integration: enphase_envoy
Integration: epion
Integration: esphome
Integration: evohome
Integration: flexit_bacnet
Integration: flipr
Integration: foscam
Integration: fritzbox
Integration: fronius
Integration: frontend
Integration: fujitsu_anywair
Integration: garages_amsterdam
Integration: gardena_bluetooth
Integration: gdacs
Integration: generic
Integration: geonetnz_quakes
Integration: geonetnz_volcano
Integration: geo_json_events
Integration: geo_rss_events
Integration: gios
Integration: glances
Integration: goodwe
Integration: google
Integration: google_generative_ai_conversation
Integration: govee_ble
Integration: govee_light_local
Integration: hive
Integration: hko
Integration: holiday
Integration: homekit_controller
Integration: homematicip_cloud
Integration: homewizard
Integration: honeywell
Integration: hp_ilo
Integration: html5
Integration: http
Integration: hue
Integration: hunterdouglas_powerview
Integration: husqvarna_automower
Integration: huum
Integration: hydrawise
Integration: iammeter
Integration: ibeacon
Integration: idasen_desk
Integration: ign_sismologia
Integration: image_upload
Integration: indianamichiganpower
Integration: insteon
Integration: ipp
Integration: islamic_prayer_times
Integration: justnimbus
Integration: jvc_projector
Integration: kef
Integration: kentuckypower
Integration: keymitt_ble
Integration: knx
Integration: kostal_plenticore
Integration: krispol
Integration: lamarzocco
Integration: ld2410_ble
Integration: leaone
Integration: led_ble
Integration: life360
Integration: lifx
Integration: linear_garage_door
Integration: litejet
Integration: local_calendar
Integration: local_todo
Integration: loqed
Integration: luci
Integration: lupusec
Integration: lutron
Integration: lutron_caseta
Integration: madeco
Integration: matrix
Integration: matter
Integration: melcloud
Integration: meteoclimatic
Integration: meteo_france
Integration: metoffice
Integration: microbees
Integration: mill
Integration: minecraft_server
Integration: modbus
Integration: mopeka
Integration: motionmount
Integration: motion_blinds
Integration: myuplink
Integration: nam
Integration: netatmo
Integration: nexia
Integration: nextbus
Integration: nextcloud
Integration: nextdns
Integration: nibe_heatpump
Integration: nmap_tracker
Integration: notion
Integration: nsw_rural_fire_service_feed
Integration: nuki
Integration: numato
Integration: nws
Integration: openai_conversation
Integration: openerz
Integration: open_meteo
Integration: opower
Integration: orvibo
Integration: osoenergy
Integration: otbr
Integration: ourgroceries
Integration: overkiz
Integration: p1_monitor
Integration: pegel_online
Integration: permobil
Integration: plex
Integration: plugwise
Integration: powerwall
Integration: private_ble_device
Integration: proxy
Integration: prusalink
Integration: psoklahoma
Integration: pure_energie
Integration: pvoutput
Integration: qingping
Integration: qld_bushfire
Integration: qrcode
Integration: rabbitair
Integration: rachio
Integration: radio_browser
Integration: rainbird
Integration: rainforest_raven
Integration: rainmachine
Integration: rdw
Integration: recorder
Integration: refoss
Integration: renault
Integration: renson
Integration: reolink
Integration: rflink
Integration: rfxtrx
Integration: ridwell
Integration: ring
Integration: risco
Integration: roborock
Integration: roku
Integration: romy
Integration: roomba
Integration: roon
Integration: route53
Integration: rova
Integration: samsam
Integration: samsungtv
Integration: schlage
Integration: scl
Integration: screenlogic
Integration: sensibo
Integration: sensorpush
Integration: sentry
Integration: seven_segments
Integration: sfr_box
Integration: shelly
Integration: sighthound
Integration: signal_messenger
Integration: simplisafe
Integration: sleepiq
Integration: smarttub
Integration: snmp
Integration: songpal
Integration: sonos
Integration: sql
Integration: squeezebox
Integration: ssdp
Integration: subaru
Integration: suez_water
Integration: sunweg
Integration: surepetcare
Integration: swepco
Integration: swiss_public_transport
Integration: switchbot
Integration: switchbot_cloud
Integration: switcher_kis
Integration: system_bridge
Integration: tado
Integration: tailwind
Integration: tankerkoenig
Integration: tasmota
Integration: technove
Integration: tedee
Integration: temper
Integration: tensorflow
Integration: teslemetry
Integration: tessie
Integration: thermobeacon
Integration: thermopro
Integration: thread
Integration: tolo
Integration: tplink
Integration: tplink_omada
Integration: tplink_tapo
Integration: traccar
Integration: traccar_server
Integration: trafikverket_camera
Integration: trafikverket_ferry
Integration: trafikverket_train
Integration: trafikverket_weatherstation
Integration: transmission
Integration: tuya
Integration: twentemilieu
Integration: unifi
Integration: unifiprotect
Integration: unifi_direct
Integration: upnp
Integration: usgs_earthquakes_feed
Integration: v2c
Integration: vallox
Integration: valve
Integration: velbus
Integration: velux
Integration: vicare
Integration: vodafone_station
Integration: vulcan
Integration: wallbox
Integration: waqi
Integration: weatherflow_cloud
Integration: weatherkit
Integration: webmin
Integration: webostv
Integration: wemo
Integration: withings
Integration: wolflink
Integration: workday
Integration: wyoming
Integration: xiaomi_ble
Integration: yalexs_ble
Integration: yeelight
Integration: yolink
Integration: zamg
Integration: zeroconf
Integration: zha
Integration: zhong_hong
Integration: zondergas
Integration: zoneminder
Integration: zwave_js
invalid
New Integration
Python 3.10
Python 3.11
question
requirement
Requirement vanished
slot-conflict
Source Incomplete
SrcDir ${S} mismatch
TopLevelViolation
Update required
virtual
wontfix
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: onkelbeh/HomeAssistantRepository#258
No description provided.