File collisions: dev-python/whois and dev-python/python-whois (different packages, both from here) #258

New Issue

2022-02-03T07:05:06+01:00

Tatsh commented

2022-02-03 07:05:06 +01:00

I attempted to upgrade to the latest homeassistant-min which, with my USE flags, requires dev-python/whois. dev-python/python-whois is already installed on my machine. I am not sure of the USE flags pulling both in.

I would prefer to not have to pick things based on potential dependencies. If these both have to be installed, then one should be renamed and the relevant code patched.

My USE flags:

accuweather
airly
airvisual
alpha_vantage
android_ip_webcam
androidtv
bluetooth_le_tracker
cast
coronavirus
github
homekit_controller
http
kodi
myq
-plex
ring
speedtestdotnet
systemmonitor
tile
tplink
wemo
whois

Log:

* This package will overwrite one or more files that may belong to other
 * packages (see list below).
 *
 * Detected file collision(s):
 *
 *      /usr/lib/python3.9/site-packages/whois/__init__.py
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.pyc
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-1.pyc
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-2.pyc
 *
 * Searching all installed packages for file collisions...
 *
 * Press Ctrl-C to Stop
 *
 * dev-python/python-whois-0.7.3:0::HomeAssistantRepository
 *      /usr/lib/python3.9/site-packages/whois/__init__.py
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-1.pyc
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-2.pyc
 *      /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.pyc
 *
 * Package 'dev-python/whois-0.9.13' NOT merged due to file collisions.
 * If necessary, refer to your elog messages for the whole content of the
 * above message.

I attempted to upgrade to the latest `homeassistant-min` which, with my USE flags, requires `dev-python/whois`. `dev-python/python-whois` is already installed on my machine. I am not sure of the USE flags pulling both in. I would prefer to not have to pick things based on potential dependencies. If these both have to be installed, then one should be renamed and the relevant code patched. My USE flags: ``` accuweather airly airvisual alpha_vantage android_ip_webcam androidtv bluetooth_le_tracker cast coronavirus github homekit_controller http kodi myq -plex ring speedtestdotnet systemmonitor tile tplink wemo whois ``` Log: ``` * This package will overwrite one or more files that may belong to other * packages (see list below). * * Detected file collision(s): * * /usr/lib/python3.9/site-packages/whois/__init__.py * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.pyc * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-1.pyc * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-2.pyc * * Searching all installed packages for file collisions... * * Press Ctrl-C to Stop * * dev-python/python-whois-0.7.3:0::HomeAssistantRepository * /usr/lib/python3.9/site-packages/whois/__init__.py * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-1.pyc * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.opt-2.pyc * /usr/lib/python3.9/site-packages/whois/__pycache__/__init__.cpython-39.pyc * * Package 'dev-python/whois-0.9.13' NOT merged due to file collisions. * If necessary, refer to your elog messages for the whole content of the * above message. ```

onkelbeh added the Cleanup label 2022-02-05 10:50:50 +01:00

onkelbeh self-assigned this 2022-02-05 10:50:56 +01:00

onkelbeh commented

2022-02-05 10:51:56 +01:00

Yes, this was changed upstream, current version is ~dev-python/whois-0.9.13 (05defe259d)
In fact it's the same package with a new name. I cannot remove ~dev-python/python-whois-0.7.3 now, because it is still referenced by old versions:

homeassistant-2021.10.7.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] )
homeassistant-2021.11.5.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] )
homeassistant-2021.12.10.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] )
homeassistant-2022.2.0.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] )
homeassistant-2022.2.1.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] )
homeassistant-2022.2.2.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] )

You could (temporarily) remove homeassistant from your world file to remove the old, and install the new build.

I would leave this ticket open as a reminder to remove it.

Yes, this was changed upstream, current version is ~dev-python/whois-0.9.13 (https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/05defe259da6593f3feb4c67b0fafd1f9b4efe3c) In fact it's the same package with a new name. I cannot remove ~dev-python/python-whois-0.7.3 now, because it is still referenced by old versions: ``` homeassistant-2021.10.7.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] ) homeassistant-2021.11.5.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] ) homeassistant-2021.12.10.ebuild: whois? ( ~dev-python/python-whois-0.7.3[${PYTHON_USEDEP}] ) homeassistant-2022.2.0.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] ) homeassistant-2022.2.1.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] ) homeassistant-2022.2.2.ebuild: whois? ( ~dev-python/whois-0.9.13[${PYTHON_USEDEP}] ) ``` You could (temporarily) remove homeassistant from your world file to remove the old, and install the new build. I would leave this ticket open as a reminder to remove it.

onkelbeh commented

2022-02-05 10:59:41 +01:00

Will try to change the Ebuild to remove the old package during the Update. I have seen such operations during world updates often, but currently I do not explicitly know to trigger this behavior. I will try to forbid the old package in the new Ebuild, perhaps this helps.

Tatsh commented

2022-02-05 11:10:34 +01:00

I also would like to know how that works for my own overlay. That would be the solution to the issue.

Tatsh commented

2022-02-05 11:24:45 +01:00

I updated and I am getting errors on startup regarding hass_frontend missing.

Feb 05 05:24:02 tatsh hass[3714322]: 2022-02-05 05:24:02 ERROR (MainThread) [homeassistant.setup] Error during setup of component frontend
Feb 05 05:24:02 tatsh hass[3714322]: Traceback (most recent call last):
Feb 05 05:24:02 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/setup.py", line 227, in _async_setup_component
Feb 05 05:24:02 tatsh hass[3714322]:     result = await task
Feb 05 05:24:02 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 339, in async_setup
Feb 05 05:24:02 tatsh hass[3714322]:     root_path = _frontend_root(repo_path)
Feb 05 05:24:02 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 314, in _frontend_root
Feb 05 05:24:02 tatsh hass[3714322]:     import hass_frontend
Feb 05 05:24:02 tatsh hass[3714322]: ModuleNotFoundError: No module named 'hass_frontend'
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of map. Setup failed for dependencies: frontend
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for map: (DependencyError(...), 'Could not setup dependencies: frontend')
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of my. Setup failed for dependencies: frontend
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for my: (DependencyError(...), 'Could not setup dependencies: frontend')
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of logbook. Setup failed for dependencies: frontend
Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for logbook: (DependencyError(...), 'Could not setup dependencies: frontend')
Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 WARNING (MainThread) [homeassistant.bootstrap] Detected that frontend did not load. Activating safe mode
Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Error during setup of component frontend
Feb 05 05:24:06 tatsh hass[3714322]: Traceback (most recent call last):
Feb 05 05:24:06 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/setup.py", line 227, in _async_setup_component
Feb 05 05:24:06 tatsh hass[3714322]:     result = await task
Feb 05 05:24:06 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 339, in async_setup
Feb 05 05:24:06 tatsh hass[3714322]:     root_path = _frontend_root(repo_path)
Feb 05 05:24:06 tatsh hass[3714322]:   File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 314, in _frontend_root
Feb 05 05:24:06 tatsh hass[3714322]:     import hass_frontend
Feb 05 05:24:06 tatsh hass[3714322]: ModuleNotFoundError: No module named 'hass_frontend'
Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of safe_mode. Setup failed for dependencies: frontend
Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Setup failed for safe_mode: (DependencyError(...), 'Could not setup dependencies: frontend')

I updated and I am getting errors on startup regarding `hass_frontend` missing. ``` Feb 05 05:24:02 tatsh hass[3714322]: 2022-02-05 05:24:02 ERROR (MainThread) [homeassistant.setup] Error during setup of component frontend Feb 05 05:24:02 tatsh hass[3714322]: Traceback (most recent call last): Feb 05 05:24:02 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/setup.py", line 227, in _async_setup_component Feb 05 05:24:02 tatsh hass[3714322]: result = await task Feb 05 05:24:02 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 339, in async_setup Feb 05 05:24:02 tatsh hass[3714322]: root_path = _frontend_root(repo_path) Feb 05 05:24:02 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 314, in _frontend_root Feb 05 05:24:02 tatsh hass[3714322]: import hass_frontend Feb 05 05:24:02 tatsh hass[3714322]: ModuleNotFoundError: No module named 'hass_frontend' Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of map. Setup failed for dependencies: frontend Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for map: (DependencyError(...), 'Could not setup dependencies: frontend') Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of my. Setup failed for dependencies: frontend Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for my: (DependencyError(...), 'Could not setup dependencies: frontend') Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of logbook. Setup failed for dependencies: frontend Feb 05 05:24:03 tatsh hass[3714322]: 2022-02-05 05:24:03 ERROR (MainThread) [homeassistant.setup] Setup failed for logbook: (DependencyError(...), 'Could not setup dependencies: frontend') Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 WARNING (MainThread) [homeassistant.bootstrap] Detected that frontend did not load. Activating safe mode Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Error during setup of component frontend Feb 05 05:24:06 tatsh hass[3714322]: Traceback (most recent call last): Feb 05 05:24:06 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/setup.py", line 227, in _async_setup_component Feb 05 05:24:06 tatsh hass[3714322]: result = await task Feb 05 05:24:06 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 339, in async_setup Feb 05 05:24:06 tatsh hass[3714322]: root_path = _frontend_root(repo_path) Feb 05 05:24:06 tatsh hass[3714322]: File "/usr/lib/python3.9/site-packages/homeassistant/components/frontend/__init__.py", line 314, in _frontend_root Feb 05 05:24:06 tatsh hass[3714322]: import hass_frontend Feb 05 05:24:06 tatsh hass[3714322]: ModuleNotFoundError: No module named 'hass_frontend' Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Unable to set up dependencies of safe_mode. Setup failed for dependencies: frontend Feb 05 05:24:06 tatsh hass[3714322]: 2022-02-05 05:24:06 ERROR (MainThread) [homeassistant.setup] Setup failed for safe_mode: (DependencyError(...), 'Could not setup dependencies: frontend') ```

onkelbeh commented

2022-02-05 11:41:45 +01:00

This is my upgrade on hasstest from 2021.12.10 -> 2022.2.2,
it's the -min Ebuild with all useflags execpt systemd, ozw & tradfri turned on:

Soft-blocking in both directions seems to behave as expected:

Total: 41 packages (40 upgrades, 1 new, 1 uninstall), Size of downloads: 0 KiB
Conflict: 2 blocks (all satisfied)

Added this now in 1e9ce544cd

The 'new' whois avoids external libraries, which I appreciate, we have plenty of them, already ;-)

BTW: let me add that adopting homeassistant-2022.2.x in production is a bit early now, would give them a few more days, this is a very fat update, and they are currently receiving >150 tickets/day, 4 days in a row. My production box still is on 2021.12.10.

This is my upgrade on hasstest from `2021.12.10` -> `2022.2.2`, it's the -min Ebuild with all useflags execpt `systemd`, `ozw` & `tradfri` turned on: ![grafik](/attachments/5aad821f-fc94-4617-a1dc-f291fb8a6f86) Soft-blocking in both directions seems to behave as expected: ``` Total: 41 packages (40 upgrades, 1 new, 1 uninstall), Size of downloads: 0 KiB Conflict: 2 blocks (all satisfied) ``` Added this now in https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/1e9ce544cda07fee9598c9181fead406cc206fce The 'new' whois avoids external libraries, which I appreciate, we have plenty of them, already ;-) BTW: let me add that adopting homeassistant-2022.2.x in production is a bit early now, would give them a few more days, this is a very fat update, and they are currently receiving >150 tickets/day, 4 days in a row. My production box still is on 2021.12.10.

grafik.png

3.4 MiB

Tatsh commented

2022-02-05 11:42:44 +01:00

I'll just downgrade back to 12.10 for now.

Tatsh commented

2022-02-05 11:48:40 +01:00

Actually I cannot downgrade now because requests-2.26.0 is now out of the main tree.

emerge: there are no ebuilds to satisfy "~dev-python/requests-2.26.0[python_targets_python3_8(-)?,python_targets_python3_9(-)?,python_targets_python3_10(-)?]".
(dependency required by "app-misc/homeassistant-min-2021.12.10::HomeAssistantRepository" [ebuild])

Actually I cannot downgrade now because requests-2.26.0 is now out of the main tree. ``` emerge: there are no ebuilds to satisfy "~dev-python/requests-2.26.0[python_targets_python3_8(-)?,python_targets_python3_9(-)?,python_targets_python3_10(-)?]". (dependency required by "app-misc/homeassistant-min-2021.12.10::HomeAssistantRepository" [ebuild]) ```

onkelbeh commented

2022-02-05 12:02:45 +01:00

Yeah, confirm, this must be brand-new, 20 minutes ago this still worked. will re-add it, takes a few minutes, currently updating the git host.

onkelbeh commented

2022-02-05 12:06:19 +01:00

re-added in 560c630856

re-added in https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/560c630856a3234ec526f7ff3960c252b2deaf95

Tatsh commented

2022-02-05 12:28:21 +01:00

Missing a file:

>>> Preparing source in /var/tmp/portage/dev-python/requests-2.26.0/work/requests-2.26.0 ...
 * Applying requests-2.26.0-test.patch ...
/var/tmp/portage/dev-python/requests-2.26.0/temp/environment: line 1652: /var/tmp/portage/dev-python/requests-2.26.0/files/requests-2.26.0-test.patch: No such file or directory
/var/tmp/portage/dev-python/requests-2.26.0/temp/environment: line 1655: /var/tmp/portage/dev-python/requests-2.26.0/files/requests-2.26.0-test.patch: No such file or directory

Missing a file: ``` >>> Preparing source in /var/tmp/portage/dev-python/requests-2.26.0/work/requests-2.26.0 ... * Applying requests-2.26.0-test.patch ... /var/tmp/portage/dev-python/requests-2.26.0/temp/environment: line 1652: /var/tmp/portage/dev-python/requests-2.26.0/files/requests-2.26.0-test.patch: No such file or directory /var/tmp/portage/dev-python/requests-2.26.0/temp/environment: line 1655: /var/tmp/portage/dev-python/requests-2.26.0/files/requests-2.26.0-test.patch: No such file or directory ```

onkelbeh commented

2022-02-05 13:08:32 +01:00

Regarding the frontend issue:
in the current version of home-assistant-frontend-20220203.0, they decided to remove the SDIST on Pypi, which we used to use.

12c12,18
< SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz"
---
>
> # SDIST missing on Pypi
> #SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz"
> # use Github instead
> MY_PN="frontend"
> SRC_URI="https://github.com/home-assistant/${MY_PN}/archive/refs/tags/${PV}.tar.gz -> ${P}.tar.gz"
> S="${WORKDIR}/${MY_PN}-${PV}"

Seems there are problems with setuptools and the source from Github.
There is an open ticket at frontend: https://github.com/home-assistant/frontend/issues/11542

Regarding the frontend issue: in the current version of home-assistant-frontend-20220203.0, they decided to remove the [SDIST on Pypi](https://pypi.org/project/home-assistant-frontend/#files), which we used to use. ```diff 12c12,18 < SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz" --- > > # SDIST missing on Pypi > #SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz" > # use Github instead > MY_PN="frontend" > SRC_URI="https://github.com/home-assistant/${MY_PN}/archive/refs/tags/${PV}.tar.gz -> ${P}.tar.gz" > S="${WORKDIR}/${MY_PN}-${PV}" ``` Seems there are problems with setuptools and the source from Github. There is an open ticket at frontend: https://github.com/home-assistant/frontend/issues/11542

onkelbeh commented

2022-02-05 13:18:30 +01:00

We should split this in two tickets ;-)

looked easy, only the Ebuild was removed in https://github.com/gentoo/gentoo/commit/676df577ce9ac07480f7d6be702d23367161859b#diff-f721f8955904a02e32c432f4438fc7dea01bd8ecef896d7e35a58ec1a8b0a12d, did not look at the patches, and skipped the test, bad, resulted in a 46 packages to downgrade.
now it runs again, I copied the missing patches in 06106473f8

We should split this in two tickets ;-) looked easy, only the Ebuild was removed in https://github.com/gentoo/gentoo/commit/676df577ce9ac07480f7d6be702d23367161859b#diff-f721f8955904a02e32c432f4438fc7dea01bd8ecef896d7e35a58ec1a8b0a12d, did not look at the patches, and skipped the test, bad, resulted in a 46 packages to downgrade. now it runs again, I copied the missing patches in https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/06106473f8ccfdd1375f1675b72acd7d15760211

Tatsh commented

2022-02-05 13:26:49 +01:00

Let me know when you want me to try the latest stable version of homeassistant-min.

onkelbeh commented

2022-02-05 13:57:04 +01:00

Sure, I just removed the 2022.2.2 release, we will have to fix the currently broken frontend at first.

onkelbeh commented

2022-02-05 13:57:45 +01:00

Unfortunately I have only low to no experience how to include a yarn build into Ebuilds.

Tatsh commented

2022-02-05 15:09:11 +01:00

My suggestion is not to try that. Your scripts that handle generating the ebuilds could also run the build process of the UI and package that up for distribution, and put it somewhere you can host. This would be almost the same as the releases that had SDIST.

To really make Yarn work inside of Portage the normal way would require figuring out every single Yarn package URL that needs to be downloaded by Portage (not via Yarn) and listing them in SRC_URI. Then you have to tell Yarn to pull all packages in locally from DISTDIR. Then you can run the project's build command (presumably yarn build).

This is really not ideal at all because there will be thousands of tiny tgz files downloaded just for HA frontend. And this would be, for most people, the only package that has distfiles from Yarn. It just seems pointless.

I have done a similar thing with a package that uses Nuget and we see where this is going with Go and Rust, where every ebuild just has to pull in all deps. It is not so bad for those because usually it's less than 100 files to download. We have generators/eclasses for these things that work pretty well. And it's unlikely to have 50 different versions of the same package in DISTDIR from Rust and Go, or even Nuget.

Nuget example - I am giving this example because there is not yet a Nuget eclass to help, just like there is not yet something for NPM or Yarn.

In the case of NPM/Yarn, they have given up on trying to resolve dependency trees where each package is unique in the entire tree. You can have a package that relies on some dependency that pulls in a different version of another dependency. Imagine if in Python: (x requires requests-2.26)+(y requires requests-2.27) but instead of making the dev of x resolve this issue by using a similar constraint (requests>=2.27), instead they 'silo' the y package to use its own copy of requests-2.27. It's like 'vendoring' (as it's sometimes called in Go) but on a massive scale.

Alternatively, you can use pkg_setup() (or another non-sandboxed phase) to run yarn inside S. This will download all the dependencies (but without more arguments it will not respect proxies which might be set in FETCHCOMMAND). The files would of course be cleaned out once the installation is done. I don't think Gentoo would ever accept an ebuild that does this, but it will work.

My suggestion is not to try that. Your scripts that handle generating the ebuilds could also run the build process of the UI and package that up for distribution, and put it somewhere you can host. This would be almost the same as the releases that had SDIST. To really make Yarn work inside of Portage the normal way would require figuring out every single Yarn package URL that needs to be downloaded by Portage (not via Yarn) and listing them in `SRC_URI`. Then you have to tell Yarn to pull all packages in locally from `DISTDIR`. Then you can run the project's build command (presumably `yarn build`). This is really not ideal at all because there will be *thousands* of tiny tgz files downloaded just for HA frontend. And this would be, for most people, the only package that has distfiles from Yarn. It just seems pointless. I have done a similar thing with a package that uses Nuget and we see where this is going with Go and Rust, where every ebuild just has to pull in all deps. It is not so bad for those because usually it's less than 100 files to download. We have generators/eclasses for these things that work pretty well. And it's unlikely to have 50 different versions of the same package in DISTDIR from Rust and Go, or even Nuget. [Nuget example](https://github.com/Tatsh/tatsh-overlay/blob/d618e94cfa90cb74cb55ab7154671324c492785f/games-emulation/ryujinx/ryujinx-1.1.13.ebuild) - I am giving this example because there is not yet a Nuget eclass to help, just like there is not yet something for NPM or Yarn. In the case of NPM/Yarn, they have given up on trying to resolve dependency trees where each package is unique in the entire tree. You can have a package that relies on some dependency that pulls in a different version of another dependency. Imagine if in Python: (x requires requests-2.26)+(y requires requests-2.27) but instead of making the dev of x resolve this issue by using a similar constraint (`requests>=2.27`), instead they 'silo' the y package to use its own copy of requests-2.27. It's like 'vendoring' (as it's sometimes called in Go) but on a massive scale. Alternatively, you can use `pkg_setup()` (or another non-sandboxed phase) to run `yarn` inside `S`. This will download all the dependencies (but without more arguments it will not respect proxies which might be set in `FETCHCOMMAND`). The files would of course be cleaned out once the installation is done. I don't think Gentoo would ever accept an ebuild that does this, but it will work.

onkelbeh commented

2022-02-05 17:12:31 +01:00

Yewah, thanks for the thoughts. We all know that npm does not mix well with portage. During compilation of the stuff for the git server updates, I was playing around, looks like I underestimated the volume of 'home-assistant-frontend', my biggest machine spends over 25 minutes compiling it from source, next we would need yarn & node-js installed. But it seems that I can get one working.

Now I have the part downloading in src_prepare(), it does not run in pkg_setup().

I want to know that we COULD do it, just in case.
Also I can checkout a way in downloading 'data' without having it in Manifest, we could use it in Ebuilds like dev-python/mac-vendor-lookup, where I currently have a big blob of a mac address table in the files directory.

There is a conversation going on with one of the frontend guys, looks like they are willing to change their build process and put an archive back online in the stage after the yarn build. Caused by an "out of space" issue, they can't put it back on Pypi, but I would not strain taking it from Github :-)

If they really put it back online, best way would be to let the user decide, switched by a USE flag.

On the other hand, messing around with npm in an Ebuild would also open the frontiers a bit more for Ebuilds like

zwavejs2mqtt
zigbee2mqtt
node-red

This is the experimental version:

## Copyright 1999-2022 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=8

PYTHON_COMPAT=( python3_{8..10} )

inherit distutils-r1
RESTRICT=network-sandbox

DESCRIPTION="The Home Assistant frontend"
HOMEPAGE="https://github.com/home-assistant/frontend https://pypi.org/project/home-assistant-frontend/"

# SDIST missing on Pypi
#SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz"
# use Github instead
MY_PN="frontend"
SRC_URI="https://github.com/home-assistant/${MY_PN}/archive/refs/tags/${PV}.tar.gz -> ${P}.tar.gz"
S="${WORKDIR}/${MY_PN}-${PV}"

LICENSE="Apache-2.0"
SLOT="0"
KEYWORDS="~amd64 ~arm ~arm64 ~x86 ~amd64-linux ~x86-linux"
IUSE="test"

RDEPEND="~dev-python/user-agents-2.0[${PYTHON_USEDEP}]"
BDEPEND="${REDEPEND}
	dev-python/setuptools[${PYTHON_USEDEP}]
	sys-apps/yarn
	test? (
		dev-python/nose[${PYTHON_USEDEP}]
		dev-python/pytest[${PYTHON_USEDEP}]
	)"

# Githubs source is lacking the built node modules
src_prepare() {
	yarn install || die
	eapply_user
}

src_compile() {
	./script/build_frontend || die
	default
}

Yewah, thanks for the thoughts. We all know that npm does not mix well with portage. During compilation of the stuff for the git server updates, I was playing around, looks like I underestimated the volume of 'home-assistant-frontend', my biggest machine spends over 25 minutes compiling it from source, next we would need yarn & node-js installed. But it seems that I can get one working. Now I have the part downloading in `src_prepare()`, it does not run in `pkg_setup()`. I want to know that we COULD do it, just in case. Also I can checkout a way in downloading 'data' without having it in Manifest, we could use it in Ebuilds like `dev-python/mac-vendor-lookup`, where I currently have a big blob of a mac address table in the files directory. There is a conversation going on with one of the frontend guys, looks like they are willing to change their build process and put an archive back online in the stage after the yarn build. Caused by an "out of space" issue, they can't put it back on Pypi, but I would not strain taking it from Github :-) If they really put it back online, best way would be to let the user decide, switched by a USE flag. On the other hand, messing around with npm in an Ebuild would also open the frontiers a bit more for Ebuilds like * zwavejs2mqtt * zigbee2mqtt * node-red This is the experimental version: ```bash ## Copyright 1999-2022 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 EAPI=8 PYTHON_COMPAT=( python3_{8..10} ) inherit distutils-r1 RESTRICT=network-sandbox DESCRIPTION="The Home Assistant frontend" HOMEPAGE="https://github.com/home-assistant/frontend https://pypi.org/project/home-assistant-frontend/" # SDIST missing on Pypi #SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz" # use Github instead MY_PN="frontend" SRC_URI="https://github.com/home-assistant/${MY_PN}/archive/refs/tags/${PV}.tar.gz -> ${P}.tar.gz" S="${WORKDIR}/${MY_PN}-${PV}" LICENSE="Apache-2.0" SLOT="0" KEYWORDS="~amd64 ~arm ~arm64 ~x86 ~amd64-linux ~x86-linux" IUSE="test" RDEPEND="~dev-python/user-agents-2.0[${PYTHON_USEDEP}]" BDEPEND="${REDEPEND} dev-python/setuptools[${PYTHON_USEDEP}] sys-apps/yarn test? ( dev-python/nose[${PYTHON_USEDEP}] dev-python/pytest[${PYTHON_USEDEP}] )" # Githubs source is lacking the built node modules src_prepare() { yarn install || die eapply_user } src_compile() { ./script/build_frontend || die default } ```

onkelbeh commented

2022-02-05 21:47:33 +01:00

Mark Mueller @cdce8p just filed a PR, which should provide an SDIST for the frontend from the next release.

https://github.com/home-assistant/frontend/pull/11566

For now, I'll keep the home-assistant-frontend-20220203.0-r1.ebuild as a 'dirty hack' only for this release. It has networking allowed in the sandbox, and eats up 30 minutes build time.

Mark Mueller @cdce8p just filed a PR, which should provide an SDIST for the frontend from the next release. https://github.com/home-assistant/frontend/pull/11566 For now, I'll keep the home-assistant-frontend-20220203.0-r1.ebuild as a 'dirty hack' only for this release. It has networking allowed in the sandbox, and eats up 30 minutes build time.

onkelbeh referenced this issue

2022-02-06 22:20:10 +01:00

ModuleNotFoundError: No module named 'hass_frontend' #259

onkelbeh commented

2022-02-11 07:35:03 +01:00

HA's frontend team is currently refusing to provide the SDIST again. Perhaps: spread some likes:
https://github.com/home-assistant/frontend/issues/11542
https://github.com/home-assistant/frontend/pull/11566

HA's frontend team is currently refusing to provide the SDIST again. Perhaps: spread some likes: https://github.com/home-assistant/frontend/issues/11542 https://github.com/home-assistant/frontend/pull/11566

onkelbeh commented

2022-02-11 07:35:43 +01:00

and https://github.com/onkelbeh/HomeAssistantRepository/issues/47

Tatsh commented

2022-02-11 07:46:11 +01:00

I would like to update. Is a workaround in place? If not, I will continue waiting.

onkelbeh commented

2022-02-12 15:33:47 +01:00

as you suggested, I'll provide an own build of the frontend from now on, could you give it a try?

as you suggested, I'll provide an [own build](https://git.edevau.net/onkelbeh/HomeAssistantRepository/commit/ce2d37ebe89f0d5e92430da963d0f79193481946) of the frontend from now on, could you give it a try?

onkelbeh commented

2022-02-13 09:09:08 +01:00

Seems to do it, 32 downloads per wget, no new tickets.

onkelbeh commented

2022-02-13 10:25:24 +01:00

In the meantime ~50 downloads with no new tickets.
Hope this will stay like that with the next release.
I've upgraded my pruduction system to 2022.2.6 yesterday night as well.
Let' summarize this in #259, I'll close this one.

In the meantime ~50 downloads with no new tickets. Hope this will stay like that with the next release. I've upgraded my pruduction system to 2022.2.6 yesterday night as well. Let' summarize this in #259, I'll close this one.

onkelbeh closed this issue

2022-02-13 10:25:24 +01:00

onkelbeh referenced this issue

2022-02-13 10:38:08 +01:00

ModuleNotFoundError: No module named 'hass_frontend' #259

onkelbeh referenced this issue from a commit

2023-07-10 20:24:30 +02:00

Merge pull request #258 from elliottjohnson/master

onkelbeh referenced this issue from a commit

2023-07-10 20:24:30 +02:00

Merge remote-tracking branch 'origin' into 2023.5.4

Sign in to join this conversation.