bugManeage - Bugs: bug #61026, Coreutils 8.32 known bug for arm64

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #61026: Coreutils 8.32 known bug for arm64

Submitter:  Boud Roukema <boud>
Submitted:  Tue 10 Aug 2021 09:03:43 PM UTC
   
 
Category:  Software Severity:  5 - Blocker
Item Group:  Crash Status:  Fixed
Privacy:  Public Assigned to:  boud
Open/Closed:  Closed

Jump to the original submission

Sat 08 Jan 2022 11:22:04 PM UTC, comment #10: 

Thanks for the confirmation Boud. As part of task #16103 (Software update 2022-1), Coreutils has been updated to version 9.0, so I am closing this task.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 12 Oct 2021 10:23:47 PM UTC, comment #9: 

Coreutils 9.0 does solve this bug. :) See commit aacd9b5 [1] and the discussion at https://savannah.nongnu.org/bugs/index.php?61240 .

Boud Roukema <boud>
Group Member
Tue 05 Oct 2021 10:47:29 AM UTC, comment #8: 

Great! I am very happy that the main issue was fixed with Coreutils 9.0.

When you are confident there is no problem, can you please make a new commit to only keep the relevant parts that are still necessary and make a new commit?

Probably a better place for point (4) is as a comment above the build rule for FFTW: since its good for someone who wants to use an older version of FFTW, but will confuse the reader of the commit message (and is just trying to understand the changes made in this commit).

Thanks also for reporting the situation with libbsd. It is strange (especially that it was caused by only changing coreutils). Can you define this as a separate bug so we can track down the hidden dependency later?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 04 Oct 2021 10:34:32 PM UTC, comment #7: 

Coreutils-9.0 downloaded, configured, compiled and installed OK.

So upgrading to coreutils-9.0 fixes part (1) of https://codeberg.org/boud/maneage_dev/commit/8b2cb4bffee09f9b953214e90c747063603cb3a6 , apparently without problems, though see below, and wait another 12-24 hours for confirmation...

The changes in parts (2) and (3) in the commit description, i.e. the changes in reproduce/software/make/high-level.mk , are still needed in Maneage, to handle the other two arm64 portability bugs described in the commit description.

Part (4) does not apply to current Maneage, since fftw2 (which is older than fftw-3.3.8) is not currently in Maneage (it's in elaphrocentre, since mpgrafic depends on fftw2).

The other files in the diff  3a1b96766c..8b2cb4bffe (apart from reproduce/software/make/high-level.mk ) should now be ignored. Adding the fftw target in reproduce/software/config/TARGETS.conf is better done as a private change, without being in any official Maneage commit.

I'm still rebuilding, so there could still be surprises, even though the build looks smooth so far. So far the proglib list is:

-rw-r--r-- 1 james mobian 39 Oct  5 00:00 mpc-1.1.0
-rw-r--r-- 1 james mobian 36 Oct  4 23:59 metastore-1.1.2-23-fa9170b
-rw-r--r-- 1 james mobian 14 Oct  4 23:59 gawk-5.1.0
-rw-r--r-- 1 james mobian 53 Oct  4 23:52 mpfr-4.0.2
-rw-r--r-- 1 james mobian 29 Oct  4 23:30 isl-0.18
-rw-r--r-- 1 james mobian 48 Oct  4 23:22 gmp-6.2.0
-rw-r--r-- 1 james mobian 11 Oct  4 23:22 git-2.28.0
-rw-r--r-- 1 james mobian 18 Oct  4 23:14 diffutils-3.7
-rw-r--r-- 1 james mobian  0 Oct  4 23:09 low-level-links
-rw-r--r-- 1 james mobian 12 Oct  4 23:09 curl-7.71.1
-rw-r--r-- 1 james mobian 20 Oct  4 23:07 findutils-4.7.0
-rw-r--r-- 1 james mobian 14 Oct  4 22:59 libbsd-0.10.0
-rw-r--r-- 1 james mobian 10 Oct  4 22:57 file-5.39
-rw-r--r-- 1 james mobian 15 Oct  4 22:55 which-2.21
-rw-r--r-- 1 james mobian 13 Oct  4 22:54 grep-3.4
-rw-r--r-- 1 james mobian 12 Oct  4 22:52 sed-4.8
-rw-r--r-- 1 james mobian 18 Oct  4 22:46 coreutils-9.0
-rw-r--r-- 1 james mobian 16 Oct  4 22:40 texinfo-6.7
-rw-r--r-- 1 james mobian 16 Oct  4 22:36 bash-5.0.18
-rw-r--r-- 1 james mobian 17 Oct  4 22:33 gettext-0.21
-rw-r--r-- 1 james mobian 24 Oct  4 22:00 libunistring-0.9.10
-rw-r--r-- 1 james mobian 18 Oct  4 21:49 libiconv-1.16
-rw-r--r-- 1 james mobian 18 Oct  4 21:47 pkg-config-0.29.2
-rw-r--r-- 1 james mobian 15 Oct  4 21:08 openssl-1.1.1a
-rw-r--r-- 1 james mobian 12 Oct  4 20:50 perl-5.32.0
-rw-r--r-- 1 james mobian 14 Oct  4 20:38 libxml2-2.9.9
-rw-r--r-- 1 james mobian 17 Oct  4 20:29 readline-8.0
-rw-r--r-- 1 james mobian  9 Oct  4 20:27 less-563
-rw-r--r-- 1 james mobian 18 Oct  4 20:26 libtool-2.4.6
-rw-r--r-- 1 james mobian 16 Oct  4 20:25 ncurses-6.2
-rw-r--r-- 1 james mobian 22 Oct  4 20:21 m4-1.4.18-patched
-rw-r--r-- 1 james mobian 14 Oct  4 20:16 patchelf-0.10
-rw-r--r-- 1 james mobian 13 Oct  4 20:14 tar-1.32
-rw-r--r-- 1 james mobian 15 Oct  4 20:10 xz-5.2.5
-rw-r--r-- 1 james mobian 10 Oct  4 20:08 unzip-6.0
-rw-r--r-- 1 james mobian  8 Oct  4 20:07 zip-3.0
-rw-r--r-- 1 james mobian 12 Oct  4 20:07 bzip2-1.0.6
-rw-r--r-- 1 james mobian 12 Oct  4 20:07 zlib-1.2.11
-rw-r--r-- 1 james mobian 14 Oct  4 20:05 gzip-1.10
-rw-r--r-- 1 james mobian 21 Oct  4 20:03 flock
-rw-r--r-- 1 james mobian 14 Oct  4 20:02 dash
-rw-r--r-- 1 james mobian 13 Oct  4 20:01 make
-rw-r--r-- 1 james mobian 14 Oct  4 19:58 lzip


This is based on two ./project configure commands. A Ctrl-C was used when pkg-config-0.29.2 took too long, appeared to be using one cpu at about 100%, but nothing was being written to the log file. This is quite likely a bug related to the fact that some subtle kernel issues are still being sorted out on the pinephone rather than a Maneage bug. The pinephone is not (yet) ready for the wider public... ;)

PS: Trying to remove only coreutils and then rebuild based on make rules failed, giving a bug in building libbsd :


In file included from setproctitle_ctor.c:27:
../include/bsd/unistd.h: In function 'setproctitle':
../include/bsd/unistd.h:69:2: error: expected declaration specifiers before '__printflike'
   69 |  __printflike(1, 2);
      |  ^~~~~~~~~~~~
setproctitle_ctor.c:52:2: error: parameter 'libbsd_init_func' is initialized
   52 |  __attribute__((__section__(".init_array"))) = setproctitle_init;
      |  ^~~~~~~~~~~~~
setproctitle_ctor.c:51:8: error: section attribute not allowed for 'libbsd_init_func'
   51 | void (*libbsd_init_func)(int argc, char *argv[], char *envp[])
      |        ^~~~~~~~~~~~~~~~
In file included from setproctitle_ctor.c:27:
../include/bsd/unistd.h:68:6: error: old-style parameter declarations in prototyped function definition
   68 | void setproctitle(const char *fmt, ...)
      |      ^~~~~~~~~~~~
setproctitle_ctor.c:52: error: expected '{' at end of input
   52 |  __attribute__((__section__(".init_array"))) = setproctitle_init;
      |
setproctitle_ctor.c: At top level:
cc1: note: unrecognized command-line option '-Wno-nullability-completeness' may have been intended to silence earlier diagnostics


It generally seems that removing and rebuilding one of the basic packages quite often requires a full rebuild, not just a rebuild based on basic.mk rules. In other words, there are hidden dependencies.

Boud Roukema <boud>
Group Member
Sun 03 Oct 2021 11:13:56 PM UTC, comment #6: 

A few days ago Coreutils 9.0 was released.

Can you please check if it has now implemented this fix internally (so ideally we won't have to make any change)?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 13 Aug 2021 02:10:42 PM UTC, comment #5: 

If the next official version of Coreutils isn't released by the next Maneage software update, we'll just upload our own tarball to Zenodo. It can be given version number '8.32-a3c04f8'.

Development versions are usually like this. For sxample see Gnuastro's development release tarballs. In Gnuastro I use the git-version-gen script of Gnulib which does this automatically and many other GNU packages also use it, I am not sure if Coreutils also does it, you shiuld be able to see after running 'make dist-lzip'.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 13 Aug 2021 02:00:27 PM UTC, comment #4: 

Thanks a lot for the commit Boud, I will try to look into it merge it in the next week or two.

Autoconf and Automake are highlevel (development) tools, a "basic" software should not be bootstrapped!

I meant to bootstrap the lastest commit of Coreutils on your computer, build the tarball (on your computer), upload it to a server (your webpage maybe) then use the URL option that you also mentioned. A tarball built with 'make dist-lzip' won't need bootstrapping, and thus won't need autoconf and automake ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 13 Aug 2021 01:09:48 PM UTC, comment #3: 

Autoconf and automake are built in high-level.mk . Using the host system autoconf (which might not exist) in basic.mk would be a regression in terms of reproducibility.

Alternatively, shifting autoconf into basic.mk and making it a prerequisite for coreutils might work. However, that would imply that autoconf is built with some host system tools instead of those built by coreutils. This would disrupt some of the reproducibility that we currently have. Disrupting this sequence without a lot of testing is probably not a good idea.

Boud Roukema <boud>
Group Member
Fri 13 Aug 2021 11:41:56 AM UTC, comment #2: 

Here's the fix to maneage that I propose to handle arm64:

https://codeberg.org/boud/maneage_dev/commit/8b2cb4bffee09f9b953214e90c747063603cb3a6

This should fix coreutils + ghostscript + libpng + fftw (version 3). The coreutils fix includes a comment that it's temporary and should become obsolete once coreutils-8.33 is released.



Alternatives for coreutils:

  • I don't see the point of including the tarball in the build directory, since that would not count as part of the source files of the project itself. If we started a directory reproduce/software/tarballs/, then that would violate the plain-text property of the source (without a gzip|lzip|xzip program and tar , the tarball is not readable).



coreutils-url = https://git.savannah.gnu.org/cgit/coreutils.git/snapshot/

and version.conf can have e.g.

coreutils-version = 8.32
coreutils-commit-id = a3c04f8

so that the URL for downloading is

https://git.savannah.gnu.org/cgit/coreutils.git/snapshot/coreutils-a3c04f8.tar.gz


This then requires minor changes in the build script in basic.mk  in the call to import-source and in terms of e.g. the directory name to change to. This method (as in elaphrocentre above) is generally useful for rapidly developed research-level software, of which some don't have any versioned releases at all (like Maneage ;)).

One flaw in trying to use this for coreutils is that the snapshot does not contain the script configure . The user is expected to use autoconf or autoreconf . However, since this is part of the basic cycle, I'm not sure if we want to use any dependence on autotools at this stage. Although we do have earlier installs of m4 and libtool...


Boud Roukema <boud>
Group Member
Wed 11 Aug 2021 01:41:49 PM UTC, comment #1: 

Thanks for the nice bug Boud, this is indeed a blocker!

One solution is to simply clone Coreutils and run 'make dist-lzip' and use that tarball in the project (which will contain this and any other bug fixes). Ultimately, if Coreutils 8.33 isn't out before we do a general update, we can just use the same tarball in that general Maneage update.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 10 Aug 2021 09:03:43 PM UTC, original submission:  

On an arm64 system running the Mobian derivative of Debian GNU/Linux, a project branching off commit 5a6e6e6 of upstream Maneage failed with


 56668  src/ls.c: In function 'print_dir':
 56669  src/ls.c:3026:24: error: 'SYS_getdents' undeclared (first use in this function); did you mean 'SYS_getdents64'?
 56670   3026 |           if (syscall (SYS_getdents, dirfd (dirp), NULL, 0) == -1
 56671        |                        ^~~~~~~~~~~~
 56672        |                        SYS_getdents64
 56673  src/ls.c:3026:24: note: each undeclared identifier is reported only once for each function it appears in
 56674  src/ls.c: At top level:


This is a known bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=966449 .

The chosen solution upstream is in this commit:

https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=10fcb97bd728f09d4a027eddf8ad2900f0819b0a

and Debian has a backport of the patch.

There are many small patches to coreutils listed at https://git.savannah.gnu.org/cgit/coreutils.git following release 8.32 . Debian is at 8.32-4, with 6 patches: https://sources.debian.org/src/coreutils/8.32-4/debian/patches .

I'll probably add the Debian patch in the specific project, that's easy. I could propose it for Maneage upstream, depending on what people's preferences are in terms of a manageable ;) update strategy.

This bug should disappear when 8.33 is released upstream and when Maneage updates to 8.33, whenever these two events might occur...

I put this as a 'blocker' bug, because this bug blocks the full reproduction of a reasonably light project (not needing more than 4 CPUs, a few Gb of RAM, a few Gb or disk space, as of 2021) on a smartphone.


Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-01-08 makhlaghi StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2021-10-03 makhlaghi CategoryNone Software
        Item GroupNone Crash
        StatusNone In Progress
        Assigned toNone boud

    Back to the top

    Powered by Savane 3.14-3b9d.
    Corresponding source code