bugManeage - Bugs: bug #63043, cmake build rule uses host...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #63043: cmake build rule uses host liblzma.so and other host libraries

Submitter:  Boud Roukema <boud>
Submitted:  Sat 10 Sep 2022 12:55:27 PM UTC
   
 
Category:  Software Severity:  4 - Important
Item Group:  Output not reasonable Status:  Fixed
Privacy:  Public Assigned to:  boud
Open/Closed:  Open

Jump to the original submission

Tue 20 Sep 2022 08:36:17 PM UTC, comment #8: 

Commit 52814c3 builds fully from scratch through to final verify.tex checksums on CentOS 7 and Debian 11.5. Fine by me to put this in the core branch and close this (cmake) bug.

https://codeberg.org/boud/maneage_dev/commit/52814c33554cd03a3ce54d8e25bed30688201cbc

Boud Roukema <boud>
Group Member
Thu 15 Sep 2022 04:36:23 PM UTC, comment #7: 

Thanks a lot for the complete analysis of the situation! I hadn't noticed the 'git clone' parts! Probably they are used when they decided to update those software and build new CMake tarballs, not when building from a pre-built tarball.

> My guess is that cmake will only do de_compression, not compression.


I agree! Probably as far as we are concerned (in Maneage's software building), those libraries won't be necessary at all (we de-compress the tarball before calling CMake). It probably needs these because CMake is a very non-modular and monolithic software: probably it also generates tarballs or uploads/downloads components and needs these for those scenarios. These are most probably not things that a Maneager would need in their analysis!

> I have no idea if cmake rules can have the effect of using cmake as a binary front end to _curl.


Since it does a static build to the cURL library, I probably CMake won't be using the 'curl' binary program.

> If problems do occur, then people using packages that depend on cmake will have to re-open this bug, or open a related one and propose a more robust solution.


I agree! But I recommend that they do the second option (open a new bug) for their unique situation (generally, its not a good idea to re-vitalize closed bugs, its better to cite the closed bug in the new bug instead).

> So I'm fine with --no-system-libs and the LaTeX target something like: "CMake $(cmake-version) (with internal libraries distributed in Utilities/cm* in tarball)".


Great! Can you commit this so I merge your commit into the core branch? Even though it is a very small fix and I can do it pretty fast, its good that this be in your name (who found, contributed to the fix and confirmed the fix; in other words, done all the hard work!).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 13 Sep 2022 09:40:08 PM UTC, comment #6: 

It seems like the internal versions are in:


cmake-3.24.0/Utilities/cmbzip2
cmake-3.24.0/Utilities/cmcurl
cmake-3.24.0/Utilities/cmliblzma
cmake-3.24.0/Utilities/cmzlib


but there are also scripts for doing git clone to upstream repositories of these:


cmake-3.24.0/Utilities/Scripts/update-bzip2.bash
cmake-3.24.0/Utilities/Scripts/update-curl.bash
cmake-3.24.0/Utilities/Scripts/update-liblzma.bash
cmake-3.24.0/Utilities/Scripts/update-third-party.bash
cmake-3.24.0/Utilities/Scripts/update-zlib.bash


On Debian 11.3, using --no-system-libs and grepping the log gives:

egrep -C1 --color=always "cmliblzma" log.config.20220912.1 |wc
    622    9678  450111
egrep -C1 --color=always "git clone" log.config.20220912.1 |wc
      0       0       0


On CentOS 7,  using --no-system-libs and grepping the log gives:

egrep -C1 --color=always "cmliblzma" log.config.20220912.1 |wc
    691   10164  430116
egrep -C1 --color=always "git clone" log.config.20220912.1 |wc
      0       0       0


So it seems very much like the internal cmake Utilities/ versions are used, not versions downloaded with git clone.
The output from ldd is similar to what you (Mohammad) gave.

So then the question is whether cmake having different versions of curl and the three compression libraries to the Maneage versions of the four libraries could cause build bugs, analysis bugs, or reproducibility bugs (or others?).

My guess is that cmake will only do de_compression, not compression. I have no idea if cmake rules can have the effect of using cmake as a binary front end to _curl.

Given that the four packages are, AFAIK, quite mature, it's quite likely that back-portability should be quite good between different versions that are not too far from each other. If problems do occur, then people using packages that depend on cmake will have to re-open this bug, or open a related one and propose a more robust solution.

So I'm fine with --no-system-libs and the LaTeX target something like: "CMake $(cmake-version) (with internal libraries distributed in Utilities/cm* in tarball)".

Boud Roukema <boud>
Group Member
Sun 11 Sep 2022 08:59:06 PM UTC, comment #5: 

As you mentioned, one of the reasons that I had initially added the '--system-*' options was to be sure about the versions of these libraries in the final reporting of the used software by CMake.

But since CMake's design is problematic if the change suggested before fixes teh problem, we can go ahead with this (letting CMake use its internal versions), and simply say something this in the LaTeX target: "CMake $(cmake-version) (with internal libraries distributed in tarball)".

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 11 Sep 2022 08:54:36 PM UTC, comment #4: 

I just tried building CMake by removing all those '--system-*' options, and instead completely disabled any system libs by adding '--no-system-libs' instead.

In the build commands, I noticed that the build commands contain '-DLZMA_API_STATIC' option, and in the end, and in the full log, I couldn't find any '/usr/lib/*.so' in any of the linking commands!

In the end, I could also confirm that it had done a static build to its internal libz, liblzma, libcurl and libbz2 (its not in the libraries it now links to, see the P.S.).

Can you try this on your CentOS 7?


P.S. List of libraries that CMake links with when it is built with '--no-system-libs':


$ ldd .local/bin/cmake
        libgcc_s.so.1 => /BDIR/software/installed/lib/libgcc_s.so.1 (0x00007f5fa675f000)
        libiconv.so.2 => /BDIR/software/installed/lib/libiconv.so.2 (0x00007f5fa6919000)
        libssl.so.3 => /BDIR/software/installed/lib/libssl.so.3 (0x00007f5fa6868000)
        libcrypto.so.3 => /BDIR/software/installed/lib/libcrypto.so.3 (0x00007f5fa6200000)
        libstdc++.so.6 => /BDIR/software/installed/lib/libstdc++.so.6 (0x00007f5fa5e00000)
        libz.so.1 => /BDIR/software/installed/lib/libz.so.1 (0x00007f5fa6742000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f5fa6780000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f5fa5c19000)
        linux-vdso.so.1 (0x00007ffdc496b000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f5fa7576000)


Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 11 Sep 2022 03:26:58 PM UTC, comment #3: 

I had a look in my build logs for CMake (it has always been bothering us in Maneage from the day it was inserted! Anytime someone asks me what is the worst software you have confronted, I always immediately say CMake! It doesn't even need thinking! But any way, let's get to the main point...).

When I searched for `liblzma.so`, it only shows my host's '/usr/lib/liblzma.so'! In fact, this happens for almost the libraries that it links against! The only one that it uses Maneage's own build is 'libcurl.so'!

However, when I was searching Maneage's installed library in the logs, I noticed something interesting: besides 'libcurl', it only occurs in the RPATH command ('-Wl,-rpath-link=/BDIR/software/installed/lib').

So it actually links with my operating system's 'liblzma.so', 'libz.so' and etc, but it only put's Maneage's library location in RPATH! Therefore, the reason I don't get a crash is that my operating system's liblzma.so and libz.so and etc are compatible with Maneage's.

This leak had indeed passed through our fingers and we had missed it! This is just bad design on CMake's side!

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 10 Sep 2022 11:05:11 PM UTC, comment #2: 

Compilation of 'cmake' fails on CentOS 7, so ldd .local/bin/cmake gives No such file or directory . :P

On Debian 11.3, ldd .local/bin/cmake shows that the Maneage libraries for curl, liblzma, bz2, have been linked.

So the risk might be lower than I stated earlier: if linking succeeds, then on at least two systems, the Maneage libraries are linked, not the host ones.

Hypothesis: with the current build rules, both the host system and Maneage libraries are examined, and, if both are valid, a decision is then made to use the Maneage libraries (probably because they're listed first). However, if one is invalid (missing symbols), then there's a fatal error.

On Debian 11.3, the log includes lines like

/BUILD/software/installed/bin/g++ -liconv -O3 -DNDEBUG -Wl,-rpath-link=/BUILD/software/installed/lib -L/BUILD/software/installed/lib CMakeFiles/testAffinity.dir/testAffinity.cxx.o -o testAffinity  ../../Source/libCMakeLib.a ../../Source/kwsys/libcmsys.a ../../Utilities/std/libcmstd.a ../../Utilities/cmexpat/libcmexpat.a ../../Utilities/cmlibarchive/libarchive/libcmlibarchive.a /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/liblzma.so ../../Utilities/cmzstd/libcmzstd.a /BUILD/software/installed/lib/libbz2.so /BUILD/software/installed/lib/libcurl.so ../../Utilities/cmjsoncpp/libcmjsoncpp.a ../../Utilities/cmlibuv/libcmlibuv.a -ldl -lrt ../../Utilities/cmlibrhash/libcmlibrhash.a -lpthread


The option '-L/BUILD/software/installed/lib' is given before '/usr/lib/x86_64-linux-gnu/liblzma.so', and 'g++' may work with several iterations - first check for consistency, and later make a final decision.

In any case, it should be possible to find build rules for cmake where '/usr/lib/x86_64-linux-gnu/liblzma.so' is absent or becomes '/BUILD/software/installed/lib/liblzma.so'; that would give valid command lines for the compile instructions.

The error appears to come from something like a 'configure' command earlier, but 'cmake' doesn't use a standard 'configure' file, just './bootstrap', so it's not clear where the problem happens; there are several lines like this:

checking for -Wl,--no-as-needed... -- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so - found
-- Found LibLZMA: /usr/lib/x86_64-linux-gnu/liblzma.so (found version "5.2.5")


I'm just guessing: I'm not an expert on g++ compilation algorithms or the cmake build algorithm.

Boud Roukema <boud>
Group Member
Sat 10 Sep 2022 07:36:06 PM UTC, comment #1: 

I'll look into the build logs of CMake next time I build Maneage. But a good way to check the linking status of a program or shared library is to use 'ldd' (on GNU/Linux systems). When I run it on my current build of Maneage I get the output shown in the P.S.

As you see there, except for the C library and Linux kernel, all the other libraries it links to are in Maneage. Have you tried this on the problematic system?

P.S.

$ ldd .local/bin/cmake
        libiconv.so.2 => /BDIR/software/installed/lib/libiconv.so.2 (0x00007ff6fda9c000)
        libz.so.1 => /BDIR/software/installed/lib/libz.so.1 (0x00007ff6fda7f000)
        liblzma.so.5 => /BDIR/software/installed/lib/liblzma.so.5 (0x00007ff6fcfd8000)
        libbz2.so.1.0 => /BDIR/software/installed/lib/libbz2.so.1.0 (0x00007ff6fcf9d000)
        libcurl.so.4 => /BDIR/software/installed/lib/libcurl.so.4 (0x00007ff6fcf0f000)
        libstdc++.so.6 => /BDIR/software/installed/lib/libstdc++.so.6 (0x00007ff6fcc00000)
        libgcc_s.so.1 => /BDIR/software/installed/lib/libgcc_s.so.1 (0x00007ff6fceee000)
        libssl.so.3 => /BDIR/software/installed/lib/libssl.so.3 (0x00007ff6fc880000)
        libcrypto.so.3 => /BDIR/software/installed/lib/libcrypto.so.3 (0x00007ff6fc200000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007ff6fc931000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007ff6fcb18000)
        libmd.so.0 => /usr/lib/libmd.so.0 (0x00007ff6fda72000)
        linux-vdso.so.1 => linux-vdso.so.1 (0x00007fff41ec7000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff6fdb85000)


Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 10 Sep 2022 12:55:27 PM UTC, original submission:  

DESCRIPTION:
Maneage commit f459fc2 [1] (same as core maneage 4318670, but with the xz hack reverted), configured on CentOS 7, has a fatal error when compiling cmake, as explained in bug 62700 [0]. This is because the cmake build rule in high-level.mk says

       ./bootstrap --prefix=$(idir) --system-curl --system-zlib \
                    --system-bzip2 --system-liblzma &&          \

which (presumably) means that "system" libraries should be used, not cmake's own versions. This was inserted in commit 2a3304cfb of Maneage in Dec 2018 by Mohammad. The motivation is (most likely) to avoid cmake's choice of libraries, and instead use Maneage's versions. However, cmake's bootstrap and configure system chooses, in some cases, host system versions of these libraries instead of the Maneage versions.

This is a bug in our build of cmake on any OS, since we have all four of curl, zlib, bzip2, and liblzma built earlier in the configure pipeline. We should not use host libraries that we already have prepared within Maneage.

Bug 62700 is only one example of this cmake build bug having bad consequences; other examples may have occurred undetected, and future examples can be expected.

REPRODUCE THE BUG:
To see the error, build commit f459fc2 [1] from scratch.

Do grep /usr/lib64/liblzma\.so on the log of the build, where /usr/lib64/liblzma\.so* is your host system location of liblzma.so .

You should see something like

/BUILD/build/software/installed/bin/g++ -liconv -O3 -DNDEBUG -Wl,-rpath-link=/BUILD/build/software/installed/lib -L/BUILD/build/software/installed/lib CMakeFiles/ctresalloc.dir/CTestResourceAllocation/ctresalloc.cxx.o -o ../../bin/ctresalloc  ../../Source/libCTestLib.a ../../Source/libCMakeLib.a ../../Source/kwsys/libcmsys.a ../../Utilities/std/libcmstd.a ../../Utilities/cmexpat/libcmexpat.a ../../Utilities/cmlibarchive/libarchive/libcmlibarchive.a /usr/lib64/libz.so /usr/lib64/liblzma.so ../../Utilities/cmzstd/libcmzstd.a /usr/lib64/libbz2.so ../../Utilities/cmjsoncpp/libcmjsoncpp.a ../../Utilities/cmlibuv/libcmlibuv.a -ldl -lrt ../../Utilities/cmlibrhash/libcmlibrhash.a -lpthread /BUILD/build/software/installed/lib/libcurl.so


This particular line shows that host-system liblzma.so and libbz2.so were linked, though Maneage libcurl.so was linked. In general, this bug is likely to lead to random behaviour.

FAILED HACKS:
Commit 213eddd [2] tries adding LDFLAGS and CFLAGS to the cmake rule to try to force usage of Maneage libraries.

Commit fa98e74e [3] tries adding LDFLAGS alone to the cmake rule to try to force usage of Maneage libraries.

Both these commits fail: they still show linking to /usr/lib64/liblzma.so in the cmake build.

My guess is that some sort of usage of LDFLAGS and CFLAGS should be able to work, keeping in mind that Darwin/xnu apparently has a problem if both C_INCLUDE_FLAGS and CPPFLAGS include the same directory. The question is where to insert these into the cmake build structure.

URLS:
[0] https://savannah.nongnu.org/bugs/?62700

[1] https://codeberg.org/boud/maneage_dev/commit/f459fc2199f1a2b74702efa7e8d239c276e2ed97 (core Maneage without the xz hack)

[2] https://codeberg.org/boud/maneage_dev/commit/213eddd9cc456a2a5c272ccf2080dd6027fbd070 (LDFLAGS and CFLAGS)

[3] https://codeberg.org/boud/maneage_dev/commit/fa98e74e2313026198125566c57b20ad9cbb53a5 (LDFLAGS)

Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-09-15 makhlaghi StatusNone Fixed
        Assigned toNone boud

    Back to the top

    Powered by Savane 3.14-3b9d.
    Corresponding source code