bugManeage - Bugs: bug #58263, Openssl/libgit2 compile bug

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #58263: Openssl/libgit2 compile bug

Submitter:  Boud Roukema <boud>
Submitted:  Mon 27 Apr 2020 10:25:06 PM UTC
   
 
Category:  Software Severity:  4 - Important
Item Group:  Crash Status:  Fixed
Privacy:  Public Assigned to:  boud
Open/Closed:  Closed

Jump to the original submission

Fri 01 May 2020 08:11:54 PM UTC, comment #14: 

Thanks for the check Raul, it has been merged with the core Maneage branch and I am closing the bug (for now).

Later, when we want to change the version to 1.2.x, I had added a link to this bug to check if this is necessary any more or not.

Thanks again for finding this very hard bug Boud, it will be so useful for the portability of many projects using Maneage.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 01 May 2020 12:34:54 PM UTC, comment #13: 

I was able to run it from scratch without any crash. Boud's hack can be merged into the master branch ;-)

Raul Infante-Sainz <infantesainz>
Group Member
Fri 01 May 2020 08:10:47 AM UTC, comment #12: 

Thanks a lot for the effort guys. I just tried it and it worked. Both openssl and libgit2 were installed in my macOS laptop without any problem :-)

In principle it should be fine but I will let you know in a couple of hours the result of a clean installation I am running (to be sure everything is fine).
 

comment #10:

> Great! Thanks for the check Boud. I am including Raul in this bug. When you get the chance, can you please check Boud's branch on a macOS also?
>
> In summary just these steps:
>


> git remote add boud https://codeberg.org/boud/maneage_dev.git
> git fetch boud fix_openssl_libgit2_bug
> git checkout -b opensslfix boud/fix_openssl_libgit2_bug
> rm .local/versioninfo/proglib/openssl
> rm .local/versioninfo/proglib/libgit2
> ./project configure -e


>
> If libgit2 can install with no problem, we'll merge it in the main Maneage branch.
>
> Just one thing Boud, when I fetched, I didn't get any further change in the version. Did you push the updated OpenSSL version?

Raul Infante-Sainz <infantesainz>
Group Member
Thu 30 Apr 2020 06:35:19 PM UTC, comment #11: 


> Did you push the updated OpenSSL version?


No. I didn't see much point. More recent versions of a package
should, in general, be more secure, but that's not always the
case. When I have time, I hope to propose
a task and branch with a commit for a Debian security-team
checked version of openssl.

Boud Roukema <boud>
Group Member
Thu 30 Apr 2020 12:21:23 PM UTC, comment #10: 

Great! Thanks for the check Boud. I am including Raul in this bug. When you get the chance, can you please check Boud's branch on a macOS also?

In summary just these steps:


git remote add boud https://codeberg.org/boud/maneage_dev.git
git fetch boud fix_openssl_libgit2_bug
git checkout -b opensslfix boud/fix_openssl_libgit2_bug
rm .local/versioninfo/proglib/openssl
rm .local/versioninfo/proglib/libgit2
./project configure -e


If libgit2 can install with no problem, we'll merge it in the main Maneage branch.

Just one thing Boud, when I fetched, I didn't get any further change in the version. Did you push the updated OpenSSL version?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 30 Apr 2020 08:31:24 AM UTC, comment #9: 

comment #8:

> But before we do, I just checked the OpenSSL download page and noticed that version 1.1.1g was just published 10 days ago (we are now using version 1.1.1**a**). Is this problem fixed there? You can try changing the version and checksums (and temporarily commenting the modification) to see.


That did not fix the bug:


$ cat .build/software/installed/version-info/proglib/openssl
OpenSSL 1.1.1g


but the same error occurs: software/installed/include/openssl/ec.h:274:1: error: expected declaration specifiers before 'DEPRECATEDIN_1_2_0' .

This is on commit b953465d8d3 + the 1.1.1g version number + checksum.
So no, 1.1.1g does not fix this particular bug.




Boud Roukema <boud>
Group Member
Thu 30 Apr 2020 12:48:20 AM UTC, comment #8: 

Thanks Boud, I just pulled the commit and tested this change on my system (which didn't have any problem originally) and it worked without a problem, so I think we can merge it.

But before we do, I just checked the OpenSSL download page and noticed that version 1.1.1g was just published 10 days ago (we are now using version 1.1.1**a**). Is this problem fixed there? You can try changing the version and checksums (and temporarily commenting the modification) to see. If it is, we can move to that version.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 29 Apr 2020 05:09:14 PM UTC, comment #7: 

This is an openssl bug! However, the openssl installation would only be able to detect the bug (and fail) if it had a sufficient number of make check type unit and regression tests.

For a reason I haven't been able to trace further, in OpenSSL Version 1.1.1a, which is the present version chosen in config/versions.conf and config/checksums.confopenssl/ec.h fails to include openssl/openconf.h .

I proposed a fix which should work as long as we do not update openssl. Since that's a different issue (a Debian security-team checked version would be safer than an upstream version), the following patch should work at least until then:

https://codeberg.org/boud/maneage_dev/commit/635caf11149c805bd5941aec5d061ce08a160481

There's an #ifndef test after the apparently failed include command that provides a simplified alternative to the actions of the missing inclusion in relation to deprecated code.

It seems to me that the worst effect that would presumably happen if we left this fix in place after updating openssl is that deprecated code would be included without any compile-time or run-time warning. (I haven't checked what sort of warning is provided by the macro.) This would be bad in terms of maintaining  code quality, which is especially important for something like openssl, though it should not block compilation.

Boud Roukema <boud>
Group Member
Wed 29 Apr 2020 01:33:44 PM UTC, comment #6: 

I suspect that the responsibility for the error is in the libgit2 build, not that of openssl .

I tried libgit2 v1.0.0 (more recent, likely to be more strict about deprecated functions) and v0.27.10 (older, might be less strict about deprecated functions). I got essentially the same error.


Boud Roukema <boud>
Group Member
Wed 29 Apr 2020 09:47:04 AM UTC, comment #5: 

comment #4:
 

> But what bugs me a lot is that you had already built Maneage before in the last few days and didn't have this problem, right?


I have not managed ;) to build the full Maneage system during
the last few days. See the various bugs I've been posting ;)...

Right now I'm trying to trace these lines in the openssl part of my log:

Checking whether your compiler can handle __attribute__((noreturn)) ...
Checking whether your compiler can handle __attribute__((pure)) ...
Checking whether your compiler can handle __attribute__((unused)) ...
Checking whether your compiler can handle __attribute__((deprecated)) ...


Boud Roukema <boud>
Group Member
Wed 29 Apr 2020 03:46:58 AM UTC, comment #4: 

Thanks Boud, I am just restored --host-cc. Infact while doing this, as you see in the commit message I also added a note if GCC fails on the system so the users use `--host-cc' and continue their build, while posting a bug report here :-).

If you want to manually re-build a software, it is easy: delete the corresponding file under `.local/version-info/proglib' (for standard programs and libraries) or `.local/version-info/python' (for Python packages). Note that both must be called from the top project directory.

I agree with your point on modularity. As you say the problem is the system's complexity! Sometimes we do something (usually as a test) before and forget to re-fix it, then it suddenly pops up somewhere else! Given how many times we have come up with such cases during development of Maneage, we just re-do a clean build to confirm if it is real. For example see bug #58265 that was just reported yesterday and fixed with a clean build.

About the error messages: I noticed that even though we see "error"s while installing OpenSSL, it doesn't actually stop the build process! OpenSSL successfully builds and is used by some of the software that depend on it (for example Coreutils depends on it). Some software do this, and have internal checks so if there is an error they do something else. This is a very bad practice, they should do tests in the `./configure' step! If OpenSSL's build had crashed, Maneage would never reach libgit2.

However! Looking into the errors of libgit2, I see that they are due to OpenSSL libraries (in particular `include/openssl/ec.h', complaining about `DEPRECATEDIN_1_2_0'. So I searched this and found this recent mailing list archive.

They had added the `no-deprecated' configuration option, but we haven't! However, when I searched your full log, I noticed that in a few cases you have `no-deprecated-declarations' in the compilation commands, like this:


cc -c   -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -Wall -Werror=declaration-after-statement -Werror=pointer-arith -Wextra -Wc++-compat -Wwrite-strings -O2   -DVERSION=\"1.00\" -DXS_VERSION=\"1.00\" -fPIC "-I../.."   core.c


So I re-built my OpenSSL and searched, but didn't have it anywhere in the printed compilation commands. Its strange, on my system, it didn't even built the `core.c' file (and all other files that have this option).

I thought of sharing these results while thinking on the solution.

But what bugs me a lot is that you had already built Maneage before in the last few days and didn't have this problem, right?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 28 Apr 2020 10:01:48 PM UTC, comment #3: 

The same error occurred on a clean build:


[ 26%] Building C object src/CMakeFiles/git2internal.dir/fetch.c.o
cd /tmp/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/project-build/src && /tmp/tmp_build_maneage_project/software/installe
d/bin/gcc -DHAVE_QSORT_R_GNU -DSHA1DC_CUSTOM_INCLUDE_SHA1_C=\"common.h\" -DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C=\"common.h\" -DSHA1DC_NO_STANDARD_INCLUDES=1 -D_FILE_O
FFSET_BITS=64 -I/tmp/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/project-build/src -I/tmp/tmp_build_maneage_project/softw
are/build-tmp/libgit2-0.28.2/src -I/tmp/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/include -I/tmp/tmp_build_maneage_proj
ect/software/build-tmp/libgit2-0.28.2/deps/http-parser  -D_GNU_SOURCE  -Wall -Wextra -fvisibility=hidden -fPIC -Wno-missing-field-initializers -Wstrict-aliasing -W
strict-prototypes -Wdeclaration-after-statement -Wshift-count-overflow -Wunused-const-variable -Wunused-function -Wformat -Wformat-security -Wint-conversion -Wno-d
ocumentation-deprecated-sync -g -D_DEBUG -O0   -std=gnu90 -o CMakeFiles/git2internal.dir/fetch.c.o   -c /tmp/tmp_build_maneage_project/software/build-
tmp/libgit2-0.28.2/src/fetch.c
/tmp/tmp_build_maneage_project/software/installed/bin/gcc -DHAVE_CONFIG_H -I. -I../libtiff  -I../libtiff -I/tmp/tmp_build_maneage_project
/software/installed/include  -g -O2 -Wall -W -MT tiff2bw.o -MD -MP -MF .deps/tiff2bw.Tpo -c -o tiff2bw.o tiff2bw.c
libtool: compile:  /tmp/tmp_build_maneage_project/software/installed/bin/gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I/tmp/tmp_build_maneage_proje
ct/software/installed/include -g -O2 -MT householdercomplex.lo -MD -MP -MF .deps/householdercomplex.Tpo -c householdercomplex.c -o householdercomplex.o >/dev/null
2>&1
In file included from /tmp/tmp_build_maneage_project/software/installed/include/openssl/x509.h:22,
                 from /tmp/tmp_build_maneage_project/software/installed/include/openssl/ssl.h:20,
                 from /tmp/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:16,
                 from /tmp/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/fetch.h:14,
                 from /tmp/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/fetch.c:8:
/tmp/tmp_build_maneage_project/software/installed/include/openssl/ec.h: In function 'DEPRECATEDIN_1_2_0':
/tmp/tmp_build_maneage_project/software/installed/include/openssl/ec.h:274:1: error: expected declaration specifiers before 'DEPRECATEDIN_1_2_0'
  274 | DEPRECATEDIN_1_2_0(int EC_GROUP_get_curve_GFp(const EC_GROUP *group, BIGNUM *p,
      | ^~~~~~~~~~~~~~~~~~
/tmp/tmp_build_maneage_project/software/installed/include/openssl/ec.h:407:3: error: storage class specified for parameter 'EC_builtin_curve'
  407 | } EC_builtin_curve;
      |   ^~~~~~~~~~~~~~~~
/tmp/tmp_build_maneage_project/software/installed/include/openssl/ec.h:415:30: error: expected declaration specifiers or '...' before 'EC_builtin_curve'
  415 | size_t EC_get_builtin_curves(EC_builtin_curve *r, size_t nitems);
      |                              ^~~~~~~~~~~~~~~~
/tmp/tmp_build_maneage_project/software/installed/include/openssl/ec.h:528:1: error: expected declaration specifiers before 'DEPRECATEDIN_1_2_0'
  528 | DEPRECATEDIN_1_2_0(int EC_POINT_set_affine_coordinates_GFp(const EC_GROUP *group,
      | ^~~~~~~~~~~~~~~~~~
/tmp/tmp_build_maneage_project/software/installed/include/openssl/ec.h:570:1: error: expected declaration specifiers before 'DEPRECATEDIN_1_2_0'


More system context:

 dpkg -l |egrep "gcc|libtool|glibc|libc[^a-z]"
ii  gcc                                       4:6.3.0-4                         amd64        GNU C compiler
ii  gcc-6                                     6.3.0-18+deb9u1                   amd64        GNU C compiler
ii  gcc-6-base:amd64                          6.3.0-18+deb9u1                   amd64        GCC, the GNU Compiler Collection (base package)
ii  klibc-utils                               2.0.4-9                           amd64        small utilities built with klibc for early boot
ii  libc-bin                                  2.24-11+deb9u4                    amd64        GNU C Library: Binaries
ii  libc-dev-bin                              2.24-11+deb9u4                    amd64        GNU C Library: Development binaries
ii  libc-l10n                                 2.24-11+deb9u4                    all          GNU C Library: localization files
ii  libc6:amd64                               2.24-11+deb9u4                    amd64        GNU C Library: Shared libraries
ii  libc6-dev:amd64                           2.24-11+deb9u4                    amd64        GNU C Library: Development Libraries and Header Files
ii  libgcc-6-dev:amd64                        6.3.0-18+deb9u1                   amd64        GCC support library (development files)
ii  libgcc1:amd64                             1:6.3.0-18+deb9u1                 amd64        GCC support library
ii  libklibc                                  2.0.4-9                           amd64        minimal libc subset for use with initramfs
ii  liblocale-gettext-perl                    1.07-3+b1                         amd64        module using libc functions for internationalization in Perl
ii  libltdl-dev:amd64                         2.4.6-2                           amd64        System independent dlopen wrapper for GNU libtool
ii  libltdl7:amd64                            2.4.6-2                           amd64        System independent dlopen wrapper for GNU libtool
ii  libpthread-stubs0-dev:amd64               0.3-4                             amd64        pthread stubs not provided by native libc, development files
ii  libtool                                   2.4.6-2                           all          Generic library support script
ii  linux-libc-dev:amd64                      4.9.210-1                         amd64        Linux support headers for userspace development


Boud Roukema <boud>
Group Member
Tue 28 Apr 2020 03:41:55 PM UTC, comment #2: 

This was not on a clean build.

Thanks for the gcc hint. I would definitely be in favour
of restoring --host-cc as an option.

Doing a full rebuild each time is a bit like completely
re-installing a system. The whole idea of 'make' and
modularity is that even when bugs happen, it should be
possible to not only create, but also remove, things in
a modular way (like aptitude remove foo or aptitude purge foo)
without disrupting other packages unless they depend on
the package to be removed.

I'll try a clean build...

Boud Roukema <boud>
Group Member
Tue 28 Apr 2020 02:43:56 AM UTC, comment #1: 

Did this happen on a clean build? I just deleted my whole build directory and re-built Maneage from scratch (while over [Commit git.maneage.org/project.git/commit/?id=4a53bd5e 4a53bd5e], but there was no problem and everything got built properly.

I don't see how bug #58260 can cause this problem, that can only happen for downloading the tarballs, not building the software. You had also reported successful building of the whole project (which always includes OpenSSL and libgit2), so probably its due to something temporary.

Maybe you can try re-building from scratch to see if this happens again.

A tip: in `reproduce/software/shell/configure.sh', just after we set a value for `host_cc', short for host C compiler, manually add `host_cc=1' so it doesn't spend ~1hour on building GCC (which isn't the problem here. Before we had an option `--host-cc', but I removed it. However, I am now thinking that its good for debugging other programs in a clean build to not waste time in building GCC.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 27 Apr 2020 10:25:06 PM UTC, original submission:  

DESCRIPTION: Over 10000 compile bugs for openssl and over 2000
compile bugs for libgit2.

CONTEXT: commit d474d4c, either with or without the fixes for
pkg-config, hdf5, install-tl-unx listed at
https://savannah.nongnu.org/bugs/?58260.

DETAILS:
The first 10 lines of grep -i "error:" on the log file give:


/tmp_build_maneage_project/software/installed/include/openssl/ec.h:274:1: error: expected declaration specifiers before 'DEPRECATEDIN_1_2_0'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:407:3: error: storage class specified for parameter 'EC_builtin_curve'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:415:30: error: expected declaration specifiers or '...' before 'EC_builtin_curve'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:528:1: error: expected declaration specifiers before 'DEPRECATEDIN_1_2_0'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:570:1: error: expected declaration specifiers before 'DEPRECATEDIN_1_2_0'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:774:1: error: storage class specified for parameter 'ECPKPARAMETERS_it'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:776:1: error: storage class specified for parameter 'ECPARAMETERS_it'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:1125:29: error: storage class specified for parameter 'ECDSA_SIG'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:1130:1: error: expected declaration specifiers before 'ECDSA_SIG'
/tmp_build_maneage_project/software/installed/include/openssl/ec.h:1135:21: error: expected declaration specifiers or '...' before 'ECDSA_SIG'


The same thing, but piped through grep -v openssl/ gives:


/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:25:3: error: storage class specified for parameter 'gitno_ssl'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:30:2: error: expected specifier-qualifier-list before 'gitno_ssl'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:31:3: error: storage class specified for parameter 'gitno_socket'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:39:3: error: storage class specified for parameter 'gitno_buffer'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:60:52: error: expected declaration specifiers or '...' before 'gitno_buffer'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:61:34: error: expected declaration specifiers or '...' before 'gitno_buffer'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:61:89: error: expected declaration specifiers or '...' before 'gitno_buffer'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:61:109: error: expected ';', ',' or ')' before 'void'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:62:16: error: expected declaration specifiers or '...' before 'gitno_buffer'
/tmp_build_maneage_project/software/build-tmp/libgit2-0.28.2/src/netops.h:64:20: error: expected declaration specifiers or '...' before 'gitno_buffer'


I Ctrl-C the configure before letting it finish. I didn't see
any use in seeing how many thousands of compile errors might
occur.


Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by infantesainz (Posted a comment)
  • -email is unavailable- added by makhlaghi
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-01 makhlaghi CategoryNone Software
        Item GroupNone Crash
        StatusWorks For Me Fixed
        Assigned toNone boud
        Open/ClosedOpen Closed
    2020-04-30 makhlaghi Carbon-Copy- Added -email is unavailable-
    2020-04-29 boud StatusNone Works For Me
    2020-04-28 boud CategorySoftware None
        Item GroupCrash None
        StatusIn Progress None
    2020-04-28 makhlaghi CategoryNone Software
        Item GroupNone Crash
        StatusNone In Progress

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code