taskManeage - Tasks: task #15390, Installing GNU C Library within...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15390: Installing GNU C Library within project (glibc)

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Sun 15 Sep 2019 01:00:52 PM UTC
   
 
Should Start On:  Sat 14 Sep 2019 11:00:00 PM UTC Should be Finished on:  Sat 14 Sep 2019 11:00:00 PM UTC
Category:  Software Priority:  5 - Normal
Status:  Postponed Privacy:  Public
Assigned to:  makhlaghi Percent Complete:  80%
Open/Closed:  Open Effort:  0.00

Jump to the original submission

Fri 15 Jul 2022 02:33:04 PM UTC, comment #11: 

I'm adding "(glibc)" to the task title to make it easier to find. :)

Boud Roukema <boud>
Group Member
Sun 30 Aug 2020 10:26:04 PM UTC, comment #10: 

Thanks for the explanation (reminder) Mohammad! :)

So installing gnulib as a separate package would be a low priority, I agree, since it's for package developers. So it is independent of the glibc task.

Pedantically, I see that gnulib is a package in Debian (literally speaking, it's built and installed independently) with just two executables: check-module and gnulib-tool.

The usage

gnulib-tool --import [module1 ... moduleN]

looks especially interesting for a package like cosmdist . That's now on my TODO list to try some time...

Of course,

 dpkg -L gnulib |wc
   9349    9349  411942

so you're right that the overwhelming bulk of gnulib is for choosing the particular source files (and creating the right sort of configure and Makefiles and so on) needed for a particular target package.

So this thread can return to glibc ... :)

Boud Roukema <boud>
Group Member
Thu 27 Aug 2020 08:59:07 PM UTC, comment #9: 

Thanks for bringing this up Boud. Gnulib is a very unique software: it isn't meant to be built (or installed!) independently.

Gnulib is meant to be included into the source of programs and compiled and built with the programs. I heavily rely on Gnulib in Gnuastro ;-)!

The basic idea behind Gnulib is that it contains a growing fraction of the GNU C Library components (and many more!) in a very portable manner (with internal checks for different OSs and hardware), classified as modules.

As a software author, you tell tell which modules you want in 'bootstrap.conf' (for example, see Gnuastro's 'bootstrap.conf'), and during the bootstrapping of your software (Gnulib actually gives you a 'bootstrap' script to put in your source), all the necessary Gnulib source files are copied into your project's source.

Only after bootstrapping is done do you run Autoconf and Automake to generate the './configure' and 'Makefile.in' files. Gnulib's modules have their own './configure' stage tests. In the case of Gnuastro, +95% of the configure-time tests are those of Gnulib! Then, when the user runs 'make', Gnulib will compile as a static/internal library before your program's source, and thus static-links to (gets copied inside) your library/executables. In the case of Gnuastro, its in the 'libgnuastro.so' library, which all the programs then link with dynamically.

The great result is that the software actually has all the requested components of the GNU C Library in its own source and it uses them!

So for example even when you compile Gnuastro on a macOS, it actually uses its own GNU C Library that is compiled on with it, and can provide an identical/reproducible behavior and output :-).

This is the reason that I mentioned it in the discussion of the Maneage paper: it helps so much in reproducibility and helps the software authors so much (they can rely on GNU C Library even on systems that don't have it!).

 Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 27 Aug 2020 08:05:45 PM UTC, comment #8: 

Should gnulib be a separate task to this one on glibc?


My understanding is that if we install glibc in maneage, then there would be no point in installing gnulib .

In that case, installing gnulib would only make sense if it were an easy alternative to glibc , either because it's lightweight (the source is about 62Mb for me, which is not so lightweight) or because it's easier to compile and install or is more portable than glibc.

Would there be any sense in temporarily installing gnulib either in basic.mk (since many things depend on it) or high-level-mk as an opt-in option in Maneage?

Boud Roukema <boud>
Group Member
Mon 11 May 2020 08:33:50 PM UTC, comment #7: 

Since the we started the gitlab.com/maneage group, I am maintaining my development branches there and the old repository in the links before has been deleted.

The development branch `glibc' that is discussed in the comments below is now available under the project-dev repository of that group.

 Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 13 Nov 2019 02:33:34 PM UTC, comment #6: 

In the previous comment, the last link (to a specific commit) was incorrect. Here is the correct link: https://gitlab.com/makhlaghi/reproducible-paper/commit/8340cac356c354

 Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 13 Nov 2019 12:54:19 PM UTC, comment #5: 

Thanks to the wonderful guidelines of Linux from scratch (LFS), I made some good progress in this task in the last few days (in my fork's glibc branch). It now builds a temporary GNU Binutils, GNU GCC, and GNU C Library with fixed/known versions.

It wasn't as easy as just copying and pasting from LFS because LFS assumes root privileges and in some parts this caused problems for our scenario. But I am happy that we can now build programs were we even control the C library. For example see this (checking the linked libraries with a tiny "hello world" program (notice how `libc.6.so' and `ld-linux-x86-64.so.2' are in the template's build directory, not the host system's):


$ ldd a.out
        linux-vdso.so.1 (0x00007ffd9a3be000)
        libc.so.6 => /template/build/dir/software/installed-round-2/lib/libc.so.6 (0x00007fda87f2a000)
        /template/build/dir/software/installed-round-2/lib64/ld-linux-x86-64.so.2 (0x00007fda880ea000)


But the job is not finshed yet! Because the currently built GCC was built before GNU C Library, so it still needs the host's C library. We should now build a new GCC and GNU Binutils from the template's C library so they can be fully independent of the host system.

The last commit so far in my `glibc' branch now goes all the way to the second GNU Binutils build and was tested on three different GNU/Linux systems (two of them large/remote computers) and works perfectly on all three.

 Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 08 Nov 2019 03:39:09 AM UTC, comment #4: 

Sorry, the conflict between glibc and GMP happens when `gmp.h' is in glibc's include-path (not its installation path as I mentioned before).

 Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 08 Nov 2019 03:28:42 AM UTC, comment #3: 

The previous problem was fixed by removing MPFR (which depends GMP) as a dependency of the AWK that is built for glibc. Apparently glibc has a conflict with GMP if its in it's installation path.

On this branch glibc now builds on my GNU/linux system, but while building the higher-level `basic.mk' programs I noticed two problems that still need to be fixed:

1. Many programs (like `gzip' and `tar' that were checked), need patchelf to actually link with our internal C library.

2. There was a compilation error while building `ncurses' with C++ because of the new C library.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 06 Nov 2019 09:23:38 PM UTC, comment #2: 

I forgot to mention that I am doing this work on the glibc branch of my fork of the project on Gitlab.

Also, as described in a more recent commit on that branch, the quoted error message below is from `glibc-2.30.9000-235-g807edded25' (tarball created with the most recent commit), I was just using it to confirm if this problem may have been fixed since the 2.30 release.

But as the error message shows, it still occurred in that tarball to. So probably, its an internal setting of the template that is causing it.

 Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 06 Nov 2019 09:13:41 PM UTC, comment #1: 

Work on this task has officially started. From the commit message:
"""
A separate Makefile (`glibc.mk') was added to the pipeline with the sole purpose of building the GNU C Library on GNU/Linux systems. When the host is a GNU/Linux operating system, the `configure.sh' script will first call this Makefile before calling `basic.mk'.

In `glibc.mk', all the GNU C Library dependencies are defined and installed in a separate directory structure under `$(BDIR)/software/installed-for-glibc'. They will just be used to build the GNU C Library. Only the GNU C Library will be installed in the standard place under `$(BDIR)/software/installed'.

After the GNU C Library is installed, `configure.sh' goes to the standard `basic.mk' and builds everything as before, but this time, because they have access to our build of the GNU C Library, they will link to it, not to the host's C library.

Everything goes nicely except close to the start of the build of glibc, I get an error about `__GMP_CAST' in the installed `gmp.h'. So I am not yet merging this branch into master.
"""

The full error message is this:


gcc -B/build-dir/software/installed-for-glibc/bin/ ../sysdeps/ieee754/float128/float1282mpn.c -c -std=gnu11 -fgnu89-inline  -g -O2 -Wall -Wwrite-strings -Wundef -fmerge-all-constants -frounding-math -fstack-protector-strong -Wstrict-prototypes -Wold-style-definition -fmath-errno   -fpie -fcf-protection   -ftls-model=initial-exec    -I/build-dir/software/installed-for-glibc/include   -I../include -I/build-dir/software/build-tmp/glibc-2.30.9000-235-g807edded25/glibc-build/stdlib  -I/build-dir/software/build-tmp/glibc-2.30.9000-235-g807edded25/glibc-build  -I../sysdeps/unix/sysv/linux/x86_64/64  -I../sysdeps/unix/sysv/linux/x86_64  -I../sysdeps/unix/sysv/linux/x86/include -I../sysdeps/unix/sysv/linux/x86  -I../sysdeps/x86/nptl  -I../sysdeps/unix/sysv/linux/wordsize-64  -I../sysdeps/x86_64/nptl  -I../sysdeps/unix/sysv/linux/include -I../sysdeps/unix/sysv/linux  -I../sysdeps/nptl  -I../sysdeps/pthread  -I../sysdeps/gnu  -I../sysdeps/unix/inet  -I../sysdeps/unix/sysv  -I../sysdeps/unix/x86_64  -I../sysdeps/unix  -I../sysdeps/posix  -I../sysdeps/x86_64/64  -I../sysdeps/x86_64/fpu/multiarch  -I../sysdeps/x86_64/fpu  -I../sysdeps/x86/fpu/include -I../sysdeps/x86/fpu  -I../sysdeps/x86_64/multiarch  -I../sysdeps/x86_64  -I../sysdeps/x86  -I../sysdeps/ieee754/float128  -I../sysdeps/ieee754/ldbl-96/include -I../sysdeps/ieee754/ldbl-96  -I../sysdeps/ieee754/dbl-64/wordsize-64  -I../sysdeps/ieee754/dbl-64  -I../sysdeps/ieee754/flt-32  -I../sysdeps/wordsize-64  -I../sysdeps/ieee754  -I../sysdeps/generic  -I.. -I../libio -I.  -D_LIBC_REENTRANT -include /build-dir/software/build-tmp/glibc-2.30.9000-235-g807edded25/glibc-build/libc-modules.h -DMODULE_NAME=libc -include ../include/libc-symbols.h  -DPIC     -DTOP_NAMESPACE=glibc -o /build-dir/software/build-tmp/glibc-2.30.9000-235-g807edded25/glibc-build/stdlib/float1282mpn.o -MD -MP -MF /build-dir/software/build-tmp/glibc-2.30.9000-235-g807edded25/glibc-build/stdlib/float1282mpn.o.dt -MT /build-dir/software/build-tmp/glibc-2.30.9000-235-g807edded25/glibc-build/stdlib/float1282mpn.o
In file included from divmod_1.c:27:
/build-dir/software/installed-for-glibc/include/gmp.h:322:33: error: expected declaration specifiers or '...' before '(' token
  322 | #define __GMP_CAST(type, expr)  ((type) (expr))
      |                                 ^
/build-dir/software/installed-for-glibc/include/gmp.h:1495:21: note: in expansion of macro '__GMP_CAST'
 1495 |   mpn_divrem_1 (qp, __GMP_CAST (mp_size_t, 0), np, nsize, dlimb)
      |                     ^~~~~~~~~~
divmod_1.c:43:1: note: in expansion of macro 'mpn_divmod_1'
   43 | mpn_divmod_1 (mp_ptr quot_ptr,
      | ^~~~~~~~~~~~
make[3]: *** [../o-iterator.mk;9: /build-dir/software/build-tmp/glibc-2.30.9000-235-g807edded25/glibc-build/stdlib/divmod_1.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory '/dev/shm/reproducible-paper-template/glibc-2.30.9000-235-g807edded25/stdlib'
make[2]: *** [Makefile;438: stdlib/subdir_lib] Error 2
make[2]: Leaving directory '/dev/shm/reproducible-paper-template/glibc-2.30.9000-235-g807edded25'
make[1]: *** [Makefile;12: install] Error 2
make[1]: Leaving directory '/dev/shm/reproducible-paper-template/glibc-2.30.9000-235-g807edded25/glibc-build'
make: *** [reproduce/software/make/glibc.mk:447: /build-dir/software/installed-for-glibc/version-info/proglib/glibc] Error 2


However, this same tarball built successfully on my system from scrach with no problem. When I compared the `gmp.h' of the template (top) and the one on my system (bottom), I got this:


#define __GMP_CFLAGS "-O2 -pedantic -fomit-frame-pointer -m64"
#define __GMP_CFLAGS "-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt"


But even after adding `-mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt' to CFLAGS while compiling GMP, I still got the same error.

 Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 15 Sep 2019 01:00:52 PM UTC, original submission:  

On systems that support it (not including macOS systems for example!), a much better level of control/reproducibility can be achieved if we also have a fixed version of the GNU C library.

Installing the GNU C Library itself doesn't take too long. We'll just need some bootstrapping first: build some basic necessary tools to build GNU C library, then delete all of those basic tools and re-build them with our own GNU C Library.

 Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one
bugsold binutils/new glibc incompatibility makes Maneage fail after two years (bug #66223, None)

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-09-16 boud Dependencies- bugs #66223 is dependent
    2022-07-15 boud SummaryInstalling GNU C Library within project Installing GNU C Library within project (glibc)
    2019-11-13 makhlaghi Percent Complete70% 80%
    2019-11-08 makhlaghi Assigned toNone makhlaghi
    2019-11-08 makhlaghi Percent Complete50% 70%
    2019-11-06 makhlaghi Percent Complete0% 50%

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.13-081f.
    Corresponding source code