bugManeage - Bugs: bug #61811, jinja2, extension-helpers,...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #61811: jinja2, extension-helpers, pybind11 not found, even though installed; pyyaml error

Submitter:  Raul Infante-Sainz <infantesainz>
Submitted:  Thu 13 Jan 2022 11:17:24 AM UTC
   
 
Category:  Software Severity:  3 - Normal
Item Group:  Crash Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Jump to the original submission

Sat 24 Sep 2022 07:58:54 PM UTC, comment #9: 

Commit 67daeee [0], which is one commit beyond core maneage 4318670, works for me to solve the 'pyyaml' compilation bug listed below here in bug #61811. The solution is to update from pyyaml-5.1 to pyyaml-6.0. (This seems to be independent from the other set of python bugs listed here.)

The cause of the problem is stated in the python documentation: Py_REFCNT changed in implementation to a function [1], so increasing its value with '++Py_REFCNT' is invalid, which is why the error message states 'lvalue required as increment operand'. An 'lvalue' is a 'left-hand value', i.e. that can be put in the left-hand part of an assignment operation, written '=' in C; the "value" of a function is (I guess) the address of the function in RAM, so augmenting the value of that address would be nonsense.

To see the problem for yourself, try compiling the standalone program

int Py_REFCNT(const double *o){
};

int main(void){
  double x;
  ++Py_REFCNT(&x);
}

with a C compiler. You should obtain the same error as when installing pyyaml-5.1 with python-3.10.6. Grep your log file if you didn't notice the bug before.

Commit 67daeee also adds 'setuptools_scm' dependencies for the python build rules for 'soupsieve' and 'webencodings', since otherwise these builds may fail (depending on the speed and parallel nature of compiles; these did fail for me); see also bug #63054 [2].

Astropy (for example) has to be enabled in TARGETS.conf for 67daeee to be tested.

[0] https://codeberg.org/boud/maneage_dev/commit/67daeee967fafd44e6f8199748f5cae4dd7ad5d6
 
[1] https://docs.python.org/3/c-api/structures.html#c.Py_REFCNT

[2] https://savannah.nongnu.org/bugs/?63054

Boud Roukema <boud>
Group Member
Sat 24 Sep 2022 07:43:53 PM UTC, comment #8: 

Commit ab7303c [1], which is one commit beyond core maneage 4318670, works for me to solve compilation bugs for the python packages 'extension-helpers' (as mentioned in #61811 here), and 'packaging', which is mentioned in the #61811 discussion as a dependency. The solution is to add both 'setuptools' and 'setuptools_scm' as dependencies in 'python.mk'. This solves the bug for me on Debian 11 and CentOS 7 parallel machines.

The two debian lists of dependencies of these two packages [2][3] are a strong clue that these dependencies are needed.

Astropy (for example) has to be enabled in TARGETS.conf for ab7303c to be tested.

See also: the discussion at bug #63054 [4].

[1] https://codeberg.org/boud/maneage_dev/commit/ab7303cd70e48830c22bd0f4ae7f70411afc3246

[2] https://sources.debian.org/src/python-packaging/21.3-1.1/debian/control

[3] https://salsa.debian.org/python-team/packages/extension-helpers/-/blob/master/debian/control

[4] https://savannah.nongnu.org/bugs/?63054

Boud Roukema <boud>
Group Member
Wed 21 Sep 2022 06:48:35 PM UTC, comment #7: 

DESCRIPTION: Running commit 4318670 of core maneage, with the addition of pyyaml as a target:

--- a/reproduce/software/config/TARGETS.conf
+++ b/reproduce/software/config/TARGETS.conf
@@ -39,7 +39,7 @@
 top-level-programs = gnuastro

 # Python libraries/modules.
-top-level-python =
+top-level-python = pyyaml

leads to compilation errors of the C source code of pyyaml. Independently, in a research project package, the compilation error does not appear to block a full configure of python packages, with matplotlib selected in TARGETS.conf. Thus, this is a bug in principle, though not (so far) an obvious problem in practice. Nevertheless, it should be fixed, and seems to be included in bug #61811 [0], which is why I'm adding it here.

LOG EXCERPT:

187806  /BUILD/software/installed/include/yaml.h:636:55: note: expected 'const yaml_char_t *' {aka '
const unsigned char *'} but argument is of type 'char *'
187807    636 |         const yaml_char_t *anchor, const yaml_char_t *tag, int implicit,
187808        |                                    ~~~~~~~~~~~~~~~~~~~^~~
187809  config.status: executing libtool commands
187810  ext/_yaml.c: In function '__pyx_tp_dealloc_5_yaml_CParser':
187811  ext/_yaml.c:23870:5: error: lvalue required as increment operand
187812  23870 |     ++Py_REFCNT(o);
187813        |     ^~
187814  ext/_yaml.c:23872:5: error: lvalue required as decrement operand
187815  23872 |     --Py_REFCNT(o);
187816        |     ^~
187817  ext/_yaml.c: In function '__pyx_tp_dealloc_5_yaml_CEmitter':
187818  ext/_yaml.c:24040:5: error: lvalue required as increment operand
187819  24040 |     ++Py_REFCNT(o);
187820        |     ^~
187821  ext/_yaml.c:24042:5: error: lvalue required as decrement operand
187822  24042 |     --Py_REFCNT(o);
187823        |     ^~
187824  ext/_yaml.c: In function '__Pyx_modinit_type_init_code':
187825  ext/_yaml.c:24844:25: error: 'PyTypeObject' {aka 'struct _typeobject'} has no member named 'tp_print'
187826  24844 |   __pyx_type_5_yaml_Mark.tp_print = 0;
187827        |                         ^
187828  ext/_yaml.c:24864:28: error: 'PyTypeObject' {aka 'struct _typeobject'} has no member named 'tp_print'
187829  24864 |   __pyx_type_5_yaml_CParser.tp_print = 0;
187830        |                            ^


HINTS:
It might be necessary to shift from our default python build rules to rules that use 'wheel', per the discussion at [1]. Bug [2] appears to be related, but it's not immediately clear what the bug is and whether there is a generic, rather than OS-specific, fix.

Some descriptions of 'wheel' are at [3][4].

A related guess is in [5][6] - it could be that the 'wheel' files with the 'sdist' option are not correctly built in the upstream 'pyyaml' (or 'jinja2' and so on) distributions.

Debugging this might require understanding the relation of cython to python, pyyaml and the other packages, and understanding the various updates and bugs in cython, and errors that people make when preparing python packages in relation to cython. Disclaimer: I'm just guessing here.


WHAT DID NOT WORK:

I tried

--- a/reproduce/software/make/python.mk
+++ b/reproduce/software/make/python.mk
@@ -581,7 +581,8 @@ $(ipydir)/pythran-$(pythran-version): \

 $(ipydir)/pyyaml-$(pyyaml-version): \
                  $(ibidir)/yaml-$(yaml-version) \
-                 $(ipydir)/cython-$(cython-version)
+                 $(ipydir)/cython-$(cython-version) \
+                 $(ipydir)/wheel-$(wheel-version)
        tarball=pyyaml-$(pyyaml-version).tar.gz
        $(call import-source, $(pyyaml-url), $(pyyaml-checksum))
        $(call pybuild, tar -xf, PyYAML-$(pyyaml-version), , \

but the same error occurred.

URLs:

[0] https://savannah.nongnu.org/bugs/?61811
[1] https://github.com/PyWavelets/pywt/issues/602
[2] https://github.com/yaml/pyyaml/issues/416
[3] https://realpython.com/python-wheels
[4] https://pythonwheels.com
[5] https://github.com/yaml/pyyaml/issues/179
[6] https://github.com/yaml/pyyaml/pull/188

Boud Roukema <boud>
Group Member
Wed 13 Jul 2022 10:40:08 AM UTC, comment #6: 

The same problem happens on another system (IAC's "darko" computer, run by a Maneage'd project of Zohreh Ghaffari), there is no problem with jinja2 and extension-helpers! However, an identical problem happens with 'pybind11' (as a dependency of Scipy!).

Similar to before, after the failed build of Scipy, if we simply delete '.local/version-info/python/pybind11' and re-configure Maneage, Scipy will work properly.

Later however, when Maneage was building astropy, astropy had no problem with identifying jinja2 and extension-helpers!

The problem may originate from the general way we install Python packages! Probably we aren't constraining something properly: based on the fact that it is not a reproducible bug!

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 19 Jan 2022 10:33:06 AM UTC, comment #5: 

Indeed, a non-reproducible bug is the most annoying (showing that we still haven't understood it!).

I tried repeating the 'pybuild' commands of 'packaging', 'jinja2' and 'extension-helpers' within their recipes to see if this problem can be fixed with a repeat. But I found that a simple repeated build doesn't fix the problem!

So its a dependency issue: in order for their presence to be recognized by the problematic packages (for example 'pyerfa' depends on 'packaging'), some other Python package needs to be present (which is present when the build finishes once!).

That software X has to become a dependency of the three problematic packages whose build is not recognized (be present before we build them).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 14 Jan 2022 03:27:42 PM UTC, comment #4: 

In each case, I re-ran './project configure --existing-conf' without modifying either TARGETS.conf or python.mk.

In both separate installs I had (still have):


# Programs and libraries (for Python or R modules, use respective variable).
top-level-programs  = gnuastro

# Python libraries/modules.
top-level-python    = astropy


I didn't try this with '--all-highlevel'.

If we could make the bug exactly reproducible ;), then it would be easier to solve...

Boud Roukema <boud>
Group Member
Fri 14 Jan 2022 08:50:00 AM UTC, comment #3: 

Thanks a lot for the help Boud.

I need to remove the final target of the problematic package (jinja2, etc.) to be able to finish the configure step and install all programs. Boud, do you need to remove the final targets? or it is only necessary to re-run the configure step again?


Raul Infante-Sainz <infantesainz>
Group Member
Thu 13 Jan 2022 06:30:57 PM UTC, comment #2: 

I tried this in two separate builds (not completely from scratch, but partly from scratch, and both python and the python libraries were rebuilt from scratch). Both builds were on the same many-cpu machine.


The second build, which updated many programs including gcc, compiled commit f9c8b81 (with 'astropy' enabled in TARGETS.conf) completely, without any errors. => irreproducible error.


The first build had a similar problem to the jinja2 and extension-helpers bugs. => reproducible error. Compiling 'astropy' failed with fatal compile errors of 'pyyaml':


/BUILD/software/installed/include/yaml.h:636:55: note: expected 'const yaml_char_t *' {aka 'const unsigned char *'} but argument is of type 'char *'
  636 |         const yaml_char_t *anchor, const yaml_char_t *tag, int implicit,
      |                                    ~~~~~~~~~~~~~~~~~~~^~~
ext/_yaml.c: In function '__pyx_tp_dealloc_5_yaml_CParser':
ext/_yaml.c:23870:5: error: lvalue required as increment operand
23870 |     ++Py_REFCNT(o);
      |     ^~
ext/_yaml.c:23872:5: error: lvalue required as decrement operand
23872 |     --Py_REFCNT(o);
      |     ^~
ext/_yaml.c: In function '__pyx_tp_dealloc_5_yaml_CEmitter':
ext/_yaml.c:24040:5: error: lvalue required as increment operand
24040 |     ++Py_REFCNT(o);
      |     ^~
ext/_yaml.c:24042:5: error: lvalue required as decrement operand
24042 |     --Py_REFCNT(o);
      |     ^~
...


This seems to imply a missing cython dependency. But pyyaml does depend on cython in python.mk.

A second 'configure' gave an immediate error for compiling pyerfa:


  File "/dev/shm/maneage-maneage_project_bis-boud/pyerfa-2.0.0.1/setup.py", line 12, in <module>
    import packaging.version
ModuleNotFoundError: No module named 'packaging'


But 'packaging' was already installed, and pyyerfa depends on 'packaging' in python.mk.

A third 'configure' successfully installed 'astropy'.

Possible clue, though probably not the explanation: Before and after the third configure step, I did 'find .local/ -name "*packaging*"'. Here's the difference:


$ diff -ub find.local.name.packaging.*
--- find.local.name.packaging.1 2022-01-13 17:50:27.652452401 +0100
+++ find.local.name.packaging.2 2022-01-13 17:51:48.575848367 +0100
@@ -1,4 +1,5 @@
 .local/version-info/python/packaging-21.3
+.local/version-info/.for-check-config/packaging-21.3
 .local/lib/python3.10/site-packages/packaging-21.3-py3.10.egg
 .local/lib/python3.10/site-packages/setuptools-58.3.0-py3.10.egg/pkg_resources/_vendor/packaging
 .local/lib/python3.10/site-packages/setuptools-58.3.0-py3.10.egg/setuptools/_vendor/packaging


The directory '.local/version-info/.for-check-config' is created by './project'. While that's useful for --check-config (I assume :)), I don't see how it can affect the rules in 'python.mk'.


This seems to me to be a semi-reproducible bug.

Boud Roukema <boud>
Group Member
Thu 13 Jan 2022 04:03:38 PM UTC, comment #1: 

I wonder if this is due to the same bug as https://savannah.nongnu.org/bugs/?61271 - Python build bug with curses.h in commit 775fc03 .

Boud Roukema <boud>
Group Member
Thu 13 Jan 2022 11:17:24 AM UTC, original submission:  

I am trying to run the installation of all software of Maneage from scratch. All basic software runs smoothly without any error.

For the high-level software there is something strange with Astropy. It crashes about not finding 'jinja2'. But it is a prerequisite of Astropy! If I remove the target of this package (.local/version-info/python/jinja2-3.0.3) and run the configure step again, it finishes without any problem.

The problem described above with 'jinja2' also happen with the package 'extension-helpers'.

I have been investigating a bit the origin of the problem but I found nothing. If you have any idea let me know.

Issue detected in task #16103 Software update 2022-1.

Raul Infante-Sainz <infantesainz>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by infantesainz (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-09-21 boud Summaryjinja2, extension-helpers, pybind11 not found, even though installed jinja2, extension-helpers, pybind11 not found, even though installed; pyyaml error
    2022-07-13 makhlaghi Summaryjinja2 and extension-helpers packages not found in Astropy installation jinja2, extension-helpers, pybind11 not found, even though installed

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code