bugManeage - Bugs: bug #61240, gcc 10.2.1-6 cannot compile gcc...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #61240: gcc 10.2.1-6 cannot compile gcc 10.2.0; related: unwind, iconv

Submitter:  Boud Roukema <boud>
Submitted:  Tue 28 Sep 2021 10:27:13 PM UTC
   
 
Category:  Software Severity:  5 - Blocker
Item Group:  Crash Status:  Fixed
Privacy:  Public Assigned to:  boud
Open/Closed:  Closed

Jump to the original submission

Tue 20 Sep 2022 09:53:16 PM UTC, comment #10: 

Just to clarify that --disable-multiarch was removed from basic.mk in commit 8463df97c6f of Mon Oct 4 02:51:45 2021 +0200, in core maneage, along with software updates.

Funnily enough, this fix happened, according to the time stamps, prior to the 9 Oct, 10 Oct, ..., phases of discussion of this bug (#61240).

Boud Roukema <boud>
Group Member
Tue 18 Jan 2022 12:09:29 AM UTC, comment #9: 

Thanks a lot for these confirmations Boud :-).

Since the versions of Coreutils and GCC have been updated as part of task #16103 (Software update 2022-1) and the note on "vampires" has been added as part of bug #61336 (Aarch64 = arm64 fixes); I am closing this task.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 12 Oct 2021 10:13:22 PM UTC, comment #8: 

Upgrading both the host system gcc and the Maneage gcc was necessary.

Commit aacd9b5 [1] removes the previous hack, removes '--disable-multiarch', upgrades gcc and coreutils, and Maneage should be fully reproducible from scratch through to the verification sums on the pinephone, apart from some minor caveats [2][3], which should probably go into a README.aarch64 file. So I propose aacd9b5 for merging into Maneage (the urls can be recommented once there are zenodo/akhlaghi.org copies in place). Once that's merged, I'm fine for closing this bug.

[1] https://codeberg.org/boud/maneage_dev/commit/aacd9b5edab1af73f0c99c06a7dcd7d7da1bda9b

[2] Vampires may be a problem on the pinephone/aarch64. A "vampire" is defined here as a job that is in the "R" (running) state, using nearly 95-100% of a cpu, for an extremely long time (hours), without producing any output to its log file, and is immune to being killed by the user or root with 'kill -9'. A reboot and relaunch of the './project configure --existing-conf' command is the only solution currently known (to me) for vampires. These are known to have occurred with linux-image-5.13-sunxi64 .

[3] Adding 3 Gb of swap space (man swapon, man mkswap, man dd) on the eMMC may help to reduce the chance of having errors due to the lack of RAM.

Related: https://savannah.nongnu.org/bugs/index.php?61026


Boud Roukema <boud>
Group Member
Sun 10 Oct 2021 09:31:59 PM UTC, comment #7: 


> have you tried updating Maneage's GCC to version 11.2?


As I said in comment #5:

> The bug also occurs for 10.2.1-6 compiling 11.2.0.


So 'yes'. In other words, on x86_64, the bug also occurs with --disable-multiarch for system-level 10.2.1-6 compiling 11.2.0 source. Andrew Pinski in the #gcc irc discussion considered this to not be a gcc bug - the user is expected to understand that on Debian there's no point disabling multiarch when compiling with the Debian gcc.

Do we have any strong reasons for disabling multiarch? (Apart from the obvious one that reproducibility is only intended on the single machine being used, so multiarch is not needed, but that's an argument against not needing multiarch, it's not an argument to disable multiarch since some systems have multiarch directory usage for header and library files.)

Testing on aarch64 is continuing, right now with gcc-11.2.0-8 compiling gcc-11.2.0 source code, since for combinations tested so far, there's a reproducible ICE (internal compiler error) bug that may or may not be related or prevent testing of the current bug. This may be an independent bug, although it does appear related to unwind code. Details later...

Boud Roukema <boud>
Group Member
Sun 10 Oct 2021 12:57:39 PM UTC, comment #6: 

That was a very nice analysis, thanks Boud.

Since it seems to be a version-specific issue, have you tried updating Maneage's GCC to version 11.2? As part of the updating of Maneage's software versions, this will happen anyway, so if the problem is fixed with a newer version, it may be better to avoid spending too much time on it ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 09 Oct 2021 10:51:56 PM UTC, comment #5: 

I'm restoring this to 'open' status. The workaround which is now in commit 775fc036e0 works on a few systems, but it's not a serious solution to the bug. Advice from someone or some people who understand the gcc configure/compile system properly makes more sense for getting a real fix.

Moreover, it seems that the current workaround (in commit 775fc036e0) causes a different sort of unwind-related fatal error (an "internal compiler error: segmentation fault") on aarch64 \approx arm64.

On Debian bullseye, gcc 10.2.1-6 compiling 10.2.0 has this bug independently of Maneage, so it's unlikely to be the fault of the Maneage setup. The bug also occurs for 10.2.1-6 compiling 11.2.0.

I found that removing --disable-multiarch seems to provide a neater solution to this bug than the workaround symlinks.

On the #gcc channel at irc.oftc.net I got helpful feedback from Pinskia. It seems that disabling multiarch on Debian systems will generally be likely to cause problems, because the Debian include and library directory system for gcc is set up to be multiarch. This is consistent with what I found with libunwind.

As before, I don't think that trying to force users to remove libunwind-dev from their system (especially since we don't want to force them to have root access) is an acceptable option. Unless there is a known problem with removing --disable-multiarch , then I think that removing --disable-multiarch from the gcc list of compile options would make sense as a better solution than the current symlink solution.

However, please wait a bit while I test this on aarch64...

Boud Roukema <boud>
Group Member
Fri 01 Oct 2021 02:30:04 PM UTC, comment #4: 

Thanks a lot for the deep analysis of this important problem and fixing it! I just rebuilt my Maneage GCC based on this branch and it successfully built in Arch GNU/Linux also :-). So I have merged it in Commit 775fc036e0 (I just done some edits in the commit message to be more consistent with the rest [1]), and am closing this bug :-).

[1] Using two ''s instead of `', and putting a "Configuration:" at the start of the title to let people know what part of Maneage it corresponds to. I also edited the title to be more clear. For example the bug number in the commit title, doesn't really give much "fast" information in between other commit titles. But it was good to have the link of the bug in the commit message for those interested.

Thanks again Boud :-)

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 01 Oct 2021 05:40:51 AM UTC, comment #3: 

EXPLANATION: Seems like this is now solved. Here's a difference between the two systems:

system B = with the bug:

dpkg -l |grep unwind
ii  libunwind-dev:amd64                           1.3.2-2                         amd64        library to determine the call-chain of a program - development
ii  libunwind8:amd64                              1.3.2-2                         amd64        library to determine the call-chain of a program - runtime


system OK = without the bug:

dpkg -l |grep unwind


Installing libunwind-dev (libunwind8 is a dependence of libunwind-dev) into system OK (which previously did not show the buggy behaviour) caused the bug to occur; removing it again caused the bug to disappear again. So the bug/feature is that gcc will use the system-level libunwind by default. It presumes that you want it to be used, since otherwise you wouldn't have installed it. :P  But in Maneage, the user shouldn't need root privileges, so the assumption is invalid.

What is less clear is whether the problem is only in the header files, or in both the header files and the compiled binary libraries.

PROPOSED SOLUTION: See
https://codeberg.org/boud/maneage_dev/commit/8221126409b47d224ee4990b739c1deb765228a2
for a proposed fix.

This hack sym-links the generic file libgcc/unwind-generic.h to the two directories in which a file #includes "unwind.h" or <unwind.h> , so that the gcc compilation system uses this gcc source file instead of searching for a system-level one.

This appears to solve the bug.

A minor bonus is included in the commit. While wondering if ncurses was involved, I noticed that on the system with the bug, many of the library files are created with the executable bit set, and the default 'ls' command is aliased to use the '-F' option.  Together, these could potentially lead to errors in command pipes that assume that 'ls' yields undecorated text without the addition of asterisks or escape characters for bold or coloured fonts. This bonus adds the command 'unalias ls' just before invoking 'ls', to reduce the probability of getting an unwanted asterisk or other annoying non-alphanumeric character in a string. A backup of '||true' is used in case the host system shell does not have 'alias' (I don't know if 'alias' is required in posix, though we probably should check that 'unalias' cannot do something bad).


OTHER POSSIBLE SOLUTIONS:

  • There's a gcc-10.2.0 configure option --with-system-unwind that can be set in gcc-10.2.0/configure, but this does not have a 'no' or disable option, and does not seem to anything significant for the amd64 (x86-64) architecture, so its behaviour does not match what we in Maneage might expect from its name in the context of this bug. Experimentally, setting this parameter doesn't seem to solve the bug either. (In this sense, this could be proposed as a GCC usability bug if we were sure enough; see 'MORE READING' below.)


  • Check if libunwind-dev (with slightly different names in non-Debian systems) exists in the system files, warn the reader, and strongly recommend that the system-level files be removed. This solution would violate the principle of a Maneage package only needing user-level access to the system if we included a 'sudo' type command, and leaving it as a human-readable recommendation would defeat the point of having an automated user-level script.



MORE READING:
The old discussion of the bug/feature here seems to be the most useful: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=14925 ; though https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78251 is more recent and suggests that maybe something should be proposed upstream to gcc, though I'm not sure.


BUG STATUS:
Feel to close this once either commit 8221126 or a better solution has been merged into Maneage.

Boud Roukema <boud>
Group Member
Wed 29 Sep 2021 10:36:54 PM UTC, comment #2: 

Maneage 3a1b96766 configures completely from scratch on a physically different instance of a Debian GNU/Linux bullseye amd64 system. So a subtle difference between the systems must explain the bug.

The machine without the bug has more cores, so the order of compilation might be important; and the machine without the bug has less host system packages installed, which might explain why the maneage libiconv was preferred over a non-existent system-level one.

My current suspicion lies with ncurses and libncurses++w.so.6.2 examined using ldd .

On the machine that compiled successfully:

$ ldd .local/lib/libncurses++w.so.6.2 | sed... # anonymise
        linux-vdso.so.1 (0x00007fXXXXXXXXX)
        libformw.so.6 => /BUILD/software/installed/lib/libformw.so.6 (0x00007fXXXXXXXXX)
        libmenuw.so.6 => /BUILD/software/installed/lib/libmenuw.so.6 (0x00007fXXXXXXXXX)
        libpanelw.so.6 => /BUILD/software/installed/lib/libpanelw.so.6 (0x00007fXXXXXXXXX)
        libncursesw.so.6 => /BUILD/software/installed/lib/libncursesw.so.6 (0x00007fXXXXXXXXX)
        libstdc++.so.6 => /BUILD/software/installed/lib/libstdc++.so.6 (0x00007fXXXXXXXXX)
        libgcc_s.so.1 => /BUILD/software/installed/lib/libgcc_s.so.1 (0x00007fXXXXXXXXX)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fXXXXXXXXX)
        /BUILD/software/installed/lib/libiconv.so (0x00007fXXXXXXXXX)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fXXXXXXXXX)
        /lib64/ld-linux-x86-64.so.2 (0x00007fXXXXXXXXX)


On the machine that failed:

$ ldd .local/lib/libncurses++w.so.6.2 | sed... # anonymise
        linux-vdso.so.1 (0x00007fXXXXXXXXX)
        libformw.so.6 => /BUILD/software/installed/lib/libformw.so.6 (0x00007fXXXXXXXXX)
        libmenuw.so.6 => /BUILD/software/installed/lib/libmenuw.so.6 (0x00007fXXXXXXXXX)
        libpanelw.so.6 => /BUILD/software/installed/lib/libpanelw.so.6 (0x00007fXXXXXXXXX)
        libncursesw.so.6 => /BUILD/software/installed/lib/libncursesw.so.6 (0x00007fXXXXXXXXX)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fXXXXXXXXX)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fXXXXXXXXX)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fXXXXXXXXX)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fXXXXXXXXX)
        /lib64/ld-linux-x86-64.so.2 (0x00007fXXXXXXXXX)


Making libiconv into a prerequisite for ncurses did not solve the bug; and it gave the same result from ldd.

Since ncurses has some special rules in basic.mk for linking, my guess is that the bug might be related to that. However, the machine with the successful compile has /BUILD/software/installed/lib/libiconv.so linked without any sym link.

Boud Roukema <boud>
Group Member
Tue 28 Sep 2021 10:46:14 PM UTC, comment #1: 

In the log file,


grep --color -B4 libiconv log.20210928.config.5.gcc.bug


(-B4 because the compile commands are continued over several lines) seems to show the lines where we would want to insert -L/BUILD/software/installed/lib before /BUILD/software/installed/lib/libiconv.so rather than after it, I assume. Something like a hack or addition of a .m4 file is my guess at a workaround, so that at least gcc can be compiled (by gcc).

Here is a related libiconv g++ related bug from July 2019:

https://savannah.nongnu.org/bugs/?56702

but I think that there have been a lot of iconv related changes since then.

Boud Roukema <boud>
Group Member
Tue 28 Sep 2021 10:27:13 PM UTC, original submission:  

Maneage 3a1b96766 [1] fails to build gcc in the configure stage on a Debian/bullseye (stable) system. Ironically, this means that gcc-10.2.1-6 has a bug in trying to compile gcc-10.2.0. :P

The bug appears to a known problem related to the unwind library and the choice of the appropriate iconv library (that of libc6 or that of maneage). See [2] and [3] for what appear to be the main GNU discussions, and [4] for a Darwin/xnu discussion (which leads back to [2]). The most meaningful solution appears to be [5]:


To fix this, the -liconv gettext adds should be immediately preceded by
a -L citing its location, so that other -L's introduced by other
packages don't trigger this (which would cause gdb, too, to link against
/usr/lib/libiconv.dylib and all would be well).


So as I understand it, gcc people see this as a system-level inconsistency problem (we have a system within a system), not a gcc bug. I'm not quite sure how we could or should insert a -L option into the gcc compilation rules at basic.mk or whether this would be the best way to do things consistently. While waiting for the pending and ambitious GNU C library task https://savannah.nongnu.org/task/?15390 we'll need some sort of hack, preferably that one that is the least likely to break anything or worsen reproducibility.


Full error log: [0]

I tried switching the reproduce/software/conf/* files to gcc-10.3.0 from upstream. Unsurprisingly, the same error occurs.

[0] https://cosmo.torun.pl/~boud/log.20210928.config.5.gcc.bug.gz

[1] http://git.maneage.org/project.git/commit/?id=3a1b96766c9d5c9845f8baacbbf88978db9c3f49

[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78251

[3] https://lists.gnu.org/archive/html/bug-gettext/2021-02/msg00000.html

[4] https://trac.macports.org/ticket/57198

[5] https://lists.gnu.org/archive/html/bug-gettext/2021-02/msg00001.html

Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-01-18 makhlaghi Open/ClosedOpen Closed
    2021-10-09 boud Open/ClosedClosed Open
    2021-10-01 makhlaghi CategoryNone Software
        Item GroupNone Crash
        StatusNone Fixed
        Assigned toNone boud
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code