taskManeage - Tasks: task #16268, python: bdist_wheel + installer...


You are not allowed to post comments on this tracker with your current authentication level.

task #16268: python: bdist_wheel + installer instead of egg

Submitter:  Boud Roukema <boud>
Submitted:  Sun 25 Sep 2022 03:08:34 PM UTC
Should Start On:  Sun 25 Sep 2022 12:00:00 AM UTC Should be Finished on:  Sun 25 Sep 2022 12:00:00 AM UTC
Category:  None Priority:  5 - Normal
Status:  None Privacy:  Public
Assigned to:  None Percent Complete:  0%
Open/Closed:  Open Effort:  0.00

Tue 04 Oct 2022 09:02:54 PM UTC, comment #4: 

> so it's quite possible that this bdist_wheel + installer strategy will become obsolete. I'm just guessing - I'm not a python developer

Yeah, python's ecosystem is in a bit of... flux. It's not always obvious what groups such as distros can do if "simply use pip install for everything" isn't a suitable answer -- and honestly, it never is a good answer.

My suggestion for bdist_wheel and installer is based on an understanding that the PyPA really, really want to push wheels, and this is not going away... after a lot of effort, installer actually exists these days and can serve as a baseline for installing those wheels in a simple, minimally bootstrapped, reproducible fashion. So, that's a good change to build on and helps the current situation a lot.

How to create wheel files is a lot more confusing. Pip creates them as a matter of course, and each PEP 517 build backend may have its own CLI for creating them, or recommend other methods. For setuptools, this is currently "python setup.py bdist_wheel" as a side migration from "python setup.py install". Setuptools is also deprecating setup.py install, and some other frontend commands, but it's not clear what the timeline for dropping support for that may be, if anything.

But making this change should solve real issues today, while simultaneously making it easier to switch to some other method later on, as python packaging requirements and recommendations settle. Which is good.


As noted, bootstrapping setuptools+installer itself would still involve some circularity, which is unfortunate, but the python standards groups do not seem to be focused on that use case.

Excluding those from the new installation method is reasonable and can be incrementally improved later, as the problem becomes better understood (maybe even by upstream, we can hope).

Eli Schwartz <eschwartz>
Mon 03 Oct 2022 08:05:32 AM UTC, comment #3: 

Further discussion at #python on Libera Chat led to the general feeling that python developers have multiple ideas on how to improve python packaging. After several discussions and some reading, my impression was that E Schwartz (ztrawhcse)'s recommendations for 'bdist_wheel' and 'installer' made the best sense for Maneage. The strategy based on bdist_wheel is summarised in 'python.mk' in commit d5775fa [2] so that people who go directly to python.mk understand the (current) strategy.

Commit d5775fa [2] solves bug #63054 (so that the crude hack of manually adding a specific .egg file to the python path is no longer needed) [3].

Here is the proposed new strategy:

# 2022-10-02 Build strategy for python modules as of October 2022, for
# python 3.10.6. Python build strategies are continuing to evolve. The
# dependency strategy used for Maneage, based on recommendations by
# E Schwartz (ztrawhcse) on #python (Libera Chat) to use
# python's 'installer' module [1], is as follows:
# (1) python-installer rule:  depends only on python
# (2) setuptools:  depends only on python
# (3) wheel: depends only on setuptools
# (4) 'medium' level modules:
#     any module that previously only depended on setuptools:
#     now depends on python-installer and wheel (indirectly on
#     setuptools via wheel);
# (5) 'high' level modules: depend either
#       (5a) on at least one 'medium' level module that indirectly
#            depends on (1)+(3), and thus also on (2)
#       (5b) directly depends on (1)+(2)+(3), so listed dependencies
#            are (1) python-installer + (3) wheel, and indirect is
#            (2) setuptools
# Related changes:
# * The module setuptools_scm is a 'medium' level module.
# * (1) + (2) + (3) are installed with the old method,
#   without python-installer, so
#   their build rule has the fourth (opt-in) parameter set to
# * (4) + (5) medium and high level modules are installed
#   using 'python setup.py bdist_wheel;
#          python -m installer dist/*.whl'
# [1] https://pypi.org/project/installer

If people think that this strategy is valid, or can improve it, and if it looks OK in further testing (it worked fine for me) then this task should be closable as 'done'.

The python ecosystem is going to continue developing - apparently meson is starting to be used for python builds. So in 2023 or 2024 or 2025 or so it's quite possible that this bdist_wheel + installer strategy will become obsolete. I'm just guessing - I'm not a python developer!

There are some Python Enhancement Proposals (PEPs) related to the issues of building and installing, though I didn't keep track of them. These seem to be something like RfCs for python. Most of the discussion doesn't seem to show if our reproducibility needs - minimal complexity, separate downloading and compiling/installing - are satisfied by the various 'recommended' methods.

[2] https://codeberg.org/boud/maneage_dev/commit/d5775faef8fb1d44c989c6d97be82e2536724d13

[3] https://savannah.nongnu.org/bugs/index.php?63054

Boud Roukema <boud>
Group Member
Mon 26 Sep 2022 09:50:54 AM UTC, comment #2: 

It might solve the bug #63054, but that wasn't my intention.

I think the idea is rather to replace our current 'pybuild' rule with

        if type pyhook_before &>/dev/null; then pyhook_before; fi; \
        python setup.py build; \
        python setup.py install --single-version-externally-managed; \
        if type pyhook_after &>/dev/null; then pyhook_after; fi; \
        cd ..; \

But it would be good if somebody could find some documentation - whether 'formal' documentation or a git commit description or inline source documentation - which tells us a bit more about this.

Boud Roukema <boud>
Group Member
Mon 26 Sep 2022 08:19:04 AM UTC, comment #1: 

Do you mean that with the option '--single-version-externally-managed' bug #63054 would be fixed? I could check it.

Raul Infante-Sainz <infantesainz>
Group Member
Sun 25 Sep 2022 03:08:34 PM UTC, original submission:  

On the Libera Chat channel #python, when I asked a specific question in trying to find a simple workaround for the pyerfa bug #63054 [0], Chris Warrick described 'egg' format as 'legacy packaging format that should be avoided', and instead recommended us considering an update of our build rules [1].

The key part seems to be

python setup.py install --single-version-externally-managed

although apparently a root option '--root=/SOME/PATH' could be useful too.

[0] https://savannah.nongnu.org/bugs/index.php?63054

[1] https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008

Boud Roukema <boud>
Group Member


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by eschwartz (Posted a comment)
  • -email is unavailable- added by infantesainz (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.


    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-10-03 boud Summarypython: -single-version-externally-managed instead of egg python: bdist_wheel + installer instead of egg

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code