automatic ebuild generator #4125

New Issue

fariouche · 2025-01-12T22:22:21+01:00

fariouche commented

2025-01-12 22:22:21 +01:00

Hello,

Don't if you already have seen this repo that is hosting a tool that automatically create ebuilds from pipy package.
https://github.com/jiegec/gentoo-pypi-generator

Maybe you already have your own, and i don't know if this one is better or not

Hello, Don't if you already have seen this repo that is hosting a tool that automatically create ebuilds from pipy package. https://github.com/jiegec/gentoo-pypi-generator Maybe you already have your own, and i don't know if this one is better or not

inode64 commented

2025-01-14 15:13:37 +01:00

It would be interesting to integrate it with livecheck https://github.com/Tatsh/livecheck so the overlay will be updated completely automatically

fariouche commented

2025-01-14 19:16:03 +01:00

I've tested it, and it works but the deps are without versions unfortunately. (and this is what I wanted most)
To do that the script will need to download the package from pypi, there is no other way.

So I will continue working on my script (I'm using uv pip to download and install the package in a temporary virtualenv and then use python requirements from importlib.metadata)

I've tested it, and it works but the deps are without versions unfortunately. (and this is what I wanted most) To do that the script will need to download the package from pypi, there is no other way. So I will continue working on my script (I'm using uv pip to download and install the package in a temporary virtualenv and then use python requirements from importlib.metadata)

oo-stef commented

2025-01-14 23:53:57 +01:00

Not sure if you're looking for an ebuild generator for HomeAssistant itself.
I wrote an ebuild generator (based on the ebuild generator already available in the tree), see PR #4126

Not sure if you're looking for an ebuild generator for HomeAssistant itself. I wrote an ebuild generator (based on the ebuild generator already available in the tree), see PR #4126

fariouche commented

2025-01-15 08:27:04 +01:00

not for homeassistant package because it is handling dependencies differently by pining versions and managing requirements itself making the ebuild a nightmare to create.
i do have a specific script for ha itself that parses the ha requirements of the components and the constraints file.
having a look at the pr, thanks

not for homeassistant package because it is handling dependencies differently by pining versions and managing requirements itself making the ebuild a nightmare to create. i do have a specific script for ha itself that parses the ha requirements of the components and the constraints file. having a look at the pr, thanks

fariouche commented

2025-01-15 08:41:42 +01:00

in my case i upgrade some of the packages the ha needs and are unavailable in the tree instead of creating ebuild for the missing version. If an ebuild for a package exists with same major.minor but higher micro version, i patch ha to use that version instead.

necrose99 commented

2025-01-29 17:26:28 +01:00

chatgpt at worse can help on ebuilds and or python scripts it's not 100% but it can prove useful.

https://github.com/jiegec/gentoo-pypi-generator uses custom out of tree libs and python 2 , its pretty dead...

chatgpt at worse can help on ebuilds and or python scripts it's not 100% but it can prove useful. https://github.com/jiegec/gentoo-pypi-generator uses custom out of tree libs and python 2 , its pretty dead...

bass_dr commented

2025-03-02 21:11:19 +01:00

hi, just wanted to point-out I also created a script able to fetch pypi and produce all the ebuilds required by homeassistant.

Currently it's hosted in my fork of homeassistant-core (I rebase my branch on the version that I need, so I can reuse scripts used to generated requirements_all.txt and friends)

I decided to create a separate overlay for now as I had many name clashes with ebuilds from this one, but my goal is still to share as much as possible with this repo here, and also because I wanted to get away from so many useflags in the main homeasssitant{,-min,-full}. I instead create one meta ebuild per component and expect people to install what they need, eventually providing one that depends on everything.

I think our goals are the same, to create a whole tree based on pypi's json API. What might be different is that I recursively produce every single ebuild that is needed by homeassistant, even some dependencies that requires to set specific extras, or those that are in gentoo's repository. Reason I'm doing this is because most of the time homeassistant requires a specific version but gentoo will drop it from the repository, breaking HA's dependencies.

I'll look at your script and see if there's anything we could share.

My approach is:

Rebase my script on the HA version I want to produce
Create a venv and uv pip install homeassistant, requirements_all.txt, etc. They are hints for my script to get the right version.
If my repo already contain a matching version of the package ebuild, I update it.
If gentoo or my repo contains another version of the package ebuild in gentoo, I take this one and update it.
Otherwise I start from a generic ebuild and update it.
Once this is done, I manually fix the ebuilds, there are still many manual steps to do.

My plan is to go even further by:

Improving the version of ebuilds that will be generated. Currently it's not always choosing the right one, and there are even cases where the chosen version cannot be used because HA requires a specific one. See the history in my repo, it happens less often than I first thought, but I'd still like to fix this
Add dependencies in between components. HA is giving an error while loading so there must be a way to get that.
Add more ways to hack dependencies upfront, instead of having to modify the ebuilds after, as pypi does not contain all dependencies, or the other way around.
Depend less on pip show (first implementation) and more on pypi's json api, that is much faster and more reliable in general.
Currently all exceptions are hardcoded in my script, ideally this would be configured outside
Figure out how to migrate to python 3.13 without too much pain from users, if you have ideas please share.

Script: https://github.com/bassdr/homeassistant-core/blob/gen_ebuild_tree/script/gen_ebuild_tree.py
Overlay: https://github.com/bassdr/gentoo-homeassistant

Anyway, commenting here as I think we can share efforts instead of both writing our own scripts. If you have ideas or suggestions either for my script or how I can contribute here, please share!

other related threads:

#4118
#4126

hi, just wanted to point-out I also created a script able to fetch pypi and produce all the ebuilds required by homeassistant. Currently it's hosted in my fork of homeassistant-core (I rebase my branch on the version that I need, so I can reuse scripts used to generated requirements_all.txt and friends) I decided to create a separate overlay for now as I had many name clashes with ebuilds from this one, but my goal is still to share as much as possible with this repo here, and also because I wanted to get away from so many useflags in the main homeasssitant{,-min,-full}. I instead create one meta ebuild per component and expect people to install what they need, eventually providing one that depends on everything. I think our goals are the same, to create a whole tree based on pypi's json API. What might be different is that I recursively produce every single ebuild that is needed by homeassistant, even some dependencies that requires to set specific extras, or those that are in gentoo's repository. Reason I'm doing this is because most of the time homeassistant requires a specific version but gentoo will drop it from the repository, breaking HA's dependencies. I'll look at your script and see if there's anything we could share. My approach is: - Rebase my script on the HA version I want to produce - Create a venv and uv pip install homeassistant, requirements_all.txt, etc. They are hints for my script to get the right version. - If my repo already contain a matching version of the package ebuild, I update it. - If gentoo or my repo contains another version of the package ebuild in gentoo, I take this one and update it. - Otherwise I start from a generic ebuild and update it. - Once this is done, I manually fix the ebuilds, there are still many manual steps to do. My plan is to go even further by: - Improving the version of ebuilds that will be generated. Currently it's not always choosing the right one, and there are even cases where the chosen version cannot be used because HA requires a specific one. See the history in my repo, it happens less often than I first thought, but I'd still like to fix this - Add dependencies in between components. HA is giving an error while loading so there must be a way to get that. - Add more ways to hack dependencies upfront, instead of having to modify the ebuilds after, as pypi does not contain all dependencies, or the other way around. - Depend less on pip show (first implementation) and more on pypi's json api, that is much faster and more reliable in general. - Currently all exceptions are hardcoded in my script, ideally this would be configured outside - Figure out how to migrate to python 3.13 without too much pain from users, if you have ideas please share. Script: https://github.com/bassdr/homeassistant-core/blob/gen_ebuild_tree/script/gen_ebuild_tree.py Overlay: https://github.com/bassdr/gentoo-homeassistant Anyway, commenting here as I think we can share efforts instead of both writing our own scripts. If you have ideas or suggestions either for my script or how I can contribute here, please share! other related threads: - https://git.edevau.net/onkelbeh/HomeAssistantRepository/issues/4118 - https://git.edevau.net/onkelbeh/HomeAssistantRepository/pulls/4126

onkelbeh commented

2025-03-03 12:57:47 +01:00

Thanks a lot. Your script has a lot of stuff I could use.
I' ll check it out very soon, there a lot of work to do, my production box is still on 2024.12.
I wonder how long it will take to get it running on Python 3.13 afterwards.

Thanks a lot. Your script has a lot of stuff I could use. I' ll check it out very soon, there a lot of work to do, my production box is still on 2024.12. I wonder how long it will take to get it running on Python 3.13 afterwards.

fariouche commented

2025-03-03 15:08:16 +01:00

It's very difficult to generate an ebuild for some packages (for example some that requires some libraries to build it's C/C++ code). Also, the api does not resolve the dependencies of a package. And doing it by hand will be complex I think (technically this is re-doing what pip and uv pip are doing: solving the deps). in my case I use uv pip to resolve all the dependencies for me. But then, when generating the ebuild, you need to know the distutil used and set it properly in the ebuild (poetry, setuptools etc)... I think this can be set automatically by gentoo (as the eclass already detect it properly), but there is no "auto" value...

You can always safely "upgrade" a package if the revision field is higher (this is what I do). What is annoying is that gentoo may remove a valid python package that makes the ebuild not usable anymore if generated when the package was available.

Also, one annoying aspect of HA is that it uses uv pip upgrade with usafe argument set. This make the module to upgrade its dependencies even if the package requirements are already satisfied. This is for example the case for esphome integration... it depends on aioesphomeapi.... at the time of the build, with patched HA for example, the dependency is correct. But then when HA starts for the first time, it will try to upgrade the packages to unsafe version. I've opened a bug in HA-core's github

It's very difficult to generate an ebuild for some packages (for example some that requires some libraries to build it's C/C++ code). Also, the api does not resolve the dependencies of a package. And doing it by hand will be complex I think (technically this is re-doing what pip and uv pip are doing: solving the deps). in my case I use uv pip to resolve all the dependencies for me. But then, when generating the ebuild, you need to know the distutil used and set it properly in the ebuild (poetry, setuptools etc)... I think this can be set automatically by gentoo (as the eclass already detect it properly), but there is no "auto" value... You can always safely "upgrade" a package if the revision field is higher (this is what I do). What is annoying is that gentoo may remove a valid python package that makes the ebuild not usable anymore if generated when the package was available. Also, one annoying aspect of HA is that it uses uv pip upgrade with usafe argument set. This make the module to upgrade its dependencies even if the package requirements are already satisfied. This is for example the case for esphome integration... it depends on aioesphomeapi.... at the time of the build, with patched HA for example, the dependency is correct. But then when HA starts for the first time, it will try to upgrade the packages to unsafe version. I've opened a bug in HA-core's github

Sign in to join this conversation.

6 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: onkelbeh/HomeAssistantRepository#4125