bugManeage - Bugs: bug #62700, lzma/xz compilation bug for cmake...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #62700: lzma/xz compilation bug for cmake in Maneage 27ff6f7

Submitter:  Boud Roukema <boud>
Submitted:  Sun 03 Jul 2022 05:37:02 PM UTC
   
 
Category:  Software Severity:  3 - Normal
Item Group:  Crash Status:  Postponed
Privacy:  Public Assigned to:  boud
Open/Closed:  Open

Jump to the original submission

Sat 11 Mar 2023 09:00:45 PM UTC, comment #9: 

Looks like xz 5.2.7 was released 5 months ago, and 5.4.1 was released 8 weeks ago:

https://git.tukaani.org/?p=xz.git;a=summary

So from my 10 Sep 2022 message:

> PROPOSAL:
>
> (a) revert the xz hack - i.e. merge f459fc2 [2] into core Maneage; and
> (b) wait for upstream xz to release e.g. xz-5.2.7, and then update to that in core Maneage after verification.


(a) is still recommended - merge f459fc2 into core maneage to revert the hack; and for (b), we could presumably update to 5.4.1.

I haven't tested 5.4.1.

Boud Roukema <boud>
Group Member
Tue 20 Sep 2022 08:39:33 PM UTC, comment #8: 

The time scale for xz-5.2.7 to be released should be a month or so; possbly shorter. On Maneage time scales, that's short :).

Anyone impatient, or worried about the CentOS 7 bug, can look at this patch without waiting:

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=31d80c6b261b24220776dfaeb8a04f80f80e0a24


Boud Roukema <boud>
Group Member
Sat 10 Sep 2022 05:47:23 PM UTC, comment #7: 

There are effectively two different bugs:

  • CentOS/xz bug #62700 (let's continue here) - the problem of RHEL developers using XZ library symbols in a way that they shouldn't have;


  • cmake build bug #63043 [1] - this is the build problem of cmake linking to a mix of host and Maneage libraries, that may sometimes cause fatal errors, at other times cause undetected errors, and in other cases have no effect.


PROPOSAL:

(a) revert the xz hack - i.e. merge f459fc2 [2] into core Maneage; and
(b) wait for upstream xz to release e.g. xz-5.2.7, and then update to that in core Maneage after verification.

The reason for (a) is that the easyfinder hack is not a seriously thought out and tested solution, so it's better to have configure failures on some systems like CentOS 7 for a short time rather than let a bad solution propagate further.

I think that the time scale for 5.2.7 is likely to be short, since Lasse Collins wants to avoid the CentOS 7 problem from propagating further (people might write software that is dependent on the wrong library symbols). Draft fixes can be seen on the xz experimental git branch [3], such as commit 913ddc55 'liblzma: Vaccinate against an ill patch from RHEL/CentOS 7.'


[1] https://savannah.nongnu.org/bugs/index.php?63043

[2] https://codeberg.org/boud/maneage_dev/commit/f459fc2199f1a2b74702efa7e8d239c276e2ed97

[3] xz git history

Boud Roukema <boud>
Group Member
Fri 09 Sep 2022 06:25:56 PM UTC, comment #6: 

While it is useful for xz to protect against the CentOS 7 mess, in principle, Maneage should not have detected the error, since we build both xz and cmake.

However, commit 2a3304cfb of Maneage from Dec 2018 by Mohammad [1] changes from

-       ./bootstrap --prefix=$(idir) && make && make install && \

to

+       ./bootstrap --prefix=$(idir) --system-curl --system-zlib \
+                    --system-bzip2 --system-liblzma &&          \

in the high-level.mk rule for building cmake.

This makes cmake dependent on the host system, and helped to detect the xz problem on CentOS 7. However, detecting bugs is not our primary aim :).

Is there any good reason for building cmake on the host system curl, zlib, bzip2, and liblzma?

If not, then I propose to remove all four of these --system-X options from the cmake build rule.


[1] http://git.maneage.org/project.git/commit/?id=2a3304cfb832d2f9548bb869f5a44a3412a1c3f9

Boud Roukema <boud>
Group Member
Thu 01 Sep 2022 04:18:17 PM UTC, comment #5: 

Lasse Collin, the xz developer, proposed a much cleaner fix, on the Libera Chat irc channel #tukaani, as suggested at https://tukaani.org . I'm currently testing the fix.

I've also reset the status to 'in progress', because Lasse said that the two library version numbers causing the problem are development versions, i.e. they shouldn't normally be used in any stable OS.

Boud Roukema <boud>
Group Member
Mon 08 Aug 2022 08:38:22 PM UTC, comment #4: 

The fix in https://savannah.nongnu.org/task/?16244 works for me. I think we could leave this bug open for a while though, to see if xz upstream finally comes up with any comments - e.g. let's say another month, since the main xz developer is probably on holidays.

But I've downgraded the bug status from 5-Blocker to 3-Normal, since we seem to have what is the best recommended fix - so far.


Boud Roukema <boud>
Group Member
Sun 07 Aug 2022 03:39:32 PM UTC, comment #3: 

This bug has been implemented as part of task #16244.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 15 Jul 2022 02:44:19 PM UTC, comment #2: 

See the #14991 github/easybuilders thread, which points to a pdf by Pat Riehecky [1]. It seems like the bug is not the fault of cmake in CentOS7, it's rather related to pthread/glibc in CentOS7. Our long-term solution is task 15390: install our own glibc [2] :).

Someone who understands pthread/glibc better than me can decide whether it's better to hack pthread/glibc for the CentOS7 case, or rather to stick to the xz workaround.

[1] https://lss.fnal.gov/archive/2021/slides/fermilab-slides-21-020-scd.pdf
[2] https://savannah.nongnu.org/task/?15390

Boud Roukema <boud>
Group Member
Thu 14 Jul 2022 02:40:31 PM UTC, comment #1: 

Based on discussion at 'easybuild' [1] and testing by me, it looks like the known hack is necessary and sufficient for CentOS7 systems and might not cause bugs in other systems. Commit 25539cd69c of branch 'xz_centos7_fix' of 'Maneage' [2] implements the known hack and works for me on a CentOS Linux 7.9.2009 system and a Debian GNU/Linux 11.3 system.

I've asked at xz upstream developers at #tukaani on Libera Chat [3][4] for recommendations (today 14 Jul 2022). Waiting a few days to a week or so for their response would be wise, depending on how urgently we want to fix the core branch of Maneage.

I upgraded 'severity' to 'blocker', because at least one major HPC system is running CentOS7; and I upgraded 'status' to 'works for me'.

[1] https://github.com/easybuilders/easybuild-easyconfigs/issues/14991
[2] https://codeberg.org/boud/maneage_dev/src/branch/xz_centos7_fix
[3] https://libera.chat
[4] https://tukaani.org/contact.html

Boud Roukema <boud>
Group Member
Sun 03 Jul 2022 05:37:02 PM UTC, original submission:  

SYSTEM:

Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

CentOS Linux release 7.9.2009 (Core)

initial gcc on the host: 10.2.0

env variables set in startup script sent to slurm:

LD_LIBRARY_PATH=/opt/exp_soft/local/generic/gcc/10.2.0/lib/gcc/x86_64-pc-linux-gnu/10.2.0:/opt/exp_soft/local/generic/gcc/10.2.0/lib64:/opt/exp_soft/local/generic/gcc/10.2.0/lib
LDFLAGS=-L/opt/exp_soft/local/generic/gcc/10.2.0/lib/gcc/x86_64-pc-linux-gnu/10.2.0 -L/opt/exp_soft/local/generic/gcc/10.2.0/lib64 -L/opt/exp_soft/local/generic/gcc/10.2.0/lib
CFLAGS=-I/opt/exp_soft/local/generic/gcc/10.2.0/include


maneage source: maneage-27ff6f7-git.bundle
https://git.maneage.org/project.git/commit/?id=27ff6f7d57ebdd3d3051040b982c1976f990c6e1


KEY LINES OF BUG:

...
make[3]: Leaving directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'
[100%] Built target pseudo_cpplint
make[3]: Entering directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'
Consolidate compiler generated dependencies of target pseudo_tidy
make[3]: Leaving directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'
[100%] Built target pseudo_tidy
make[3]: Entering directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'
Consolidate compiler generated dependencies of target pseudo_iwyu
make[3]: Leaving directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'
[100%] Built target pseudo_iwyu
make[3]: Entering directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'
Consolidate compiler generated dependencies of target foo
make[3]: Leaving directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'
[100%] Built target foo
make[2]: Leaving directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'
Install the project...
bin/cmake: /BUILD/software/installed/lib/liblzma.so.5: version `XZ_5.2.2' not found (required by bin/cmake)
bin/cmake: /BUILD/software/installed/lib/liblzma.so.5: version `XZ_5.1.2alpha' not found (required by bin/cmake)
make[1]: *** [Makefile:144: install] Error 1
make[1]: Leaving directory '/dev/shm/24623915-maneage-27ff6f7-git-boud/cmake-3.21.4'



FULL LOG:
full log file


CLUES:
It seems like this is a known bug for Redhat systems [1][2] since 2017.
This gives at least two puzzles:

(1) Why hasn't the fix been upstreamed to xz? (hypothesis:
because this is a one-person email-or-irc-bug-report maintained program; see
MAINTAINABILITY below)

(2) Why does maneage compiling cmake have a problem with the lzma
library api on a redhat machine but not on another machine, since
high-level software should be isolated from OS-dependent library
apis like this?

On a machine on which a merge with 27ff6f7 compiled successfully,

$ ldd .local/bin/cmake |grep lzma
liblzma.so.5 => /BUILD/software/installed/lib/liblzma.so.5 (0x00XXXXXXXXXXXXXX)


It is also possible that the patch proposed in [2] itself creates a
bug on redhat systems [3], as of a report in 2022, per Easybuild [4]
users/developers.

[1] https://stackoverflow.com/questions/47633870/rpm-lib64-liblzma-so-5-version-xz-5-1-2alpha-not-found-required-by-lib-li
[2] https://github.com/easybuilders/easybuild-easyconfigs/issues/4036
[3] https://github.com/easybuilders/easybuild-easyconfigs/issues/14991
[4] https://github.com/easybuilders/easybuild-easyconfigs

The relation between xz and lzma is not clear to me - understanding this may
help for finding a portable, safe solution.


MAINTAINABILITY of xz:

The [https://git.tukaani.org/?p=xz.git;a=blob_plain;f=README;hb=HEAD
README] file of xz says "Report your bug via email or IRC (see
Contact information below)." This means that if the main developer
is busy or ill, the community cannot fix the bugs without creating
its own fork, e.g. on a git repository.


Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-09-10 boud StatusIn Progress Postponed
    2022-09-01 boud StatusFixed In Progress
    2022-08-08 boud Open/ClosedClosed Open
    2022-08-08 boud Severity5 - Blocker 3 - Normal
    2022-08-07 makhlaghi StatusWorks For Me Fixed
        Assigned toNone boud
        Open/ClosedOpen Closed
    2022-07-14 boud Severity3 - Normal 5 - Blocker
        StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code