taskReproducible paper template - Tasks: task #15667, Install Lzip and Make before...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15667: Install Lzip and Make before calling basic.mk

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Sun 31 May 2020 04:24:03 AM UTC  
 
Should Start On:  Sat 30 May 2020 11:00:00 PM UTC Should be Finished on:  Sat 30 May 2020 11:00:00 PM UTC
Category:  Software Priority:  5 - Normal
Status:  Done Privacy:  Public
Percent Complete:  100% Assigned to:  Mohammad Akhlaghi <makhlaghi>
Open/Closed:  Closed Effort:  0.00

( Jump to the original submission)

Wed 03 Jun 2020 08:04:58 AM UTC, comment #11: 

I did and it worked perfectly!
Thanks a lot :-)

Raul Infante-Sainz <infantesainz>
Project Member
Tue 02 Jun 2020 10:40:55 PM UTC, comment #10: 

Great! It is now merged into the main Maneage branch, so I am closing this task.

It would be great if you could try this commit on a clean build ;-).

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Tue 02 Jun 2020 06:03:20 PM UTC, comment #9: 

Thanks a lot Mohammad. I am happy to say that with the addition of libiconv as a prerequisite of libunistring, it worked perfectly! So now we have everything also running in parallel on macOS :-D
To not mix things I will let you commit the fix to this minor bug. This night I will run once again (removing all tarballs), just to ensure everything is fine, but in principle we can move on! ;-)

Raul Infante-Sainz <infantesainz>
Project Member
Tue 02 Jun 2020 02:40:28 PM UTC, comment #8: 

Thanks Raul,

OK! So the error (shown below) is related to libiconv, but we now build libunistring right after tar. So probably it was busy building libiconv at the same time it was building libunistring.

Can you try adding libiconv as a dependency of libunistring and trying again? The rule for libiconv is just above the rule for libunistring.

Now that we are building in parallel on macOS, we are encountering such dependency problems for the first time, but they will be fixed soon ;-).

P.S.
Just for the record, this is the main error message:

Undefined symbols for architecture x86_64:
  "_iconv", referenced from:
      _mem_cd_iconveh_internal in striconveh.o
     (maybe you meant: _libunistring_mem_iconveha, _libunistring_mem_iconveh , _libunistring_iconveh_close , _libunistring_iconveh_open , _libunistring_str_cd_iconveh , _libunistring_str_iconveh , _libunistring_mem_cd_iconveh , _libunistring_str_iconveha )
  "_iconv_close", referenced from:
      _libunistring_iconveh_open in striconveh.o
      _libunistring_iconveh_close in striconveh.o
  "_iconv_open", referenced from:
      _libunistring_iconveh_open in striconveh.o
install ./include/openssl/evp.h -> /BDIR/software/installed/include/openssl/evp.h
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [Makefile:4215: libunistring.la] Error 1

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Tue 02 Jun 2020 08:42:31 AM UTC, comment #7: 

In macOS it crashes when installing `libunistring'. You can have a look at the attached file for all the output I obtain. Just after that, if I run again it goes up to the end. So, I think there is some dependency for `libunistring' that we are missing. But I checked that it actually doesn't depend on anything! Any idea?

(file #49200)

Raul Infante-Sainz <infantesainz>
Project Member
Mon 01 Jun 2020 08:37:29 PM UTC, comment #6: 

Thanks a lot Boud ;-). Indeed, if the PDF is built, it means that all the outputs have been successfully checked with their checksums (you can see it 'verfify.mk').

I hope it builds cleanly on macOS is well.

I need a little more time for completing a clean build on FreeBSD (as described in bug #58465), but it is certainly do-able (they are all minor).

But FreeBSD is unrelated to this task directly, so we can safely merge after confirming on macOS and we'll fix that bug soon afterwards.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Mon 01 Jun 2020 07:48:56 PM UTC, comment #5: 

Commit b840309 builds cleanly for me on x86_64 GNU/Linux. I got through to

Output written on paper.pdf (3 pages, 258148 bytes).
Transcript written on paper.log.

I haven't (yet) got to understand how to handle the verify
steps, but I didn't modify anything, so with

$ grep -v "^#" reproduce/analysis/config/verify-outputs.conf
verify-outputs = yes

I assume that the checks must have passed.

Boud Roukema <boud>
Project Member
Mon 01 Jun 2020 05:47:41 PM UTC, comment #4: 

Thanks Raul and Boud. Indeed, on some Tar implementations, Lzip isn't supported by default. I just implemented the fix in the default 'gbuild' and 'cbuild' rules that we use to build programs. It won't be a problem in other programs that have custom calls of tar, because by that point we have our own GNU Tar installed and ready to use.

So Raul, can you please pull this branch and build its most recent commit? I just finished a clean build (including download of tarballs) on my system and it worked.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Mon 01 Jun 2020 02:15:03 PM UTC, comment #3: 

Do we already have GNU tar at this stage? If yes, then we
can use the GNU style tar options (single hyphen for
one-letter options; double hyphen for long options).
If no, then things are messy
because of the complicated tar history/variations. The following
is within an already built maneage system (commit 213dac3):

maneage_project$ PATH=.local/bin tar -cv --lzip -f p.tar.lz paper.tex paper.pdf
paper.tex
paper.pdf
maneage_project$ PATH=.local/bin/ ls -l p.tar.lz paper.*
-rw-r--r-- 1 boud boud 264112 Jun  1 15:58 p.tar.lz
-rw-r--r-- 1 boud boud 258350 Jun  1 15:58 paper.pdf
-rw-r--r-- 1 boud boud  11609 Jun  1 15:58 paper.tex
maneage_project$ PATH=.local/bin/ rm -f paper.???
maneage_project$ PATH=.local/bin/ ls -l p.tar.lz paper.*
ls: cannot access 'paper.*': No such file or directory
-rw-r--r-- 1 boud boud 264112 Jun  1 15:58  p.tar.lz
maneage_project$ PATH=.local/bin tar -xv -f p.tar.lz paper.tex paper.pdf
paper.tex
paper.pdf
maneage_project$ PATH=.local/bin/ ls -l p.tar.lz paper.*
-rw-r--r-- 1 boud boud 264112 Jun  1 15:58 p.tar.lz
-rw-r--r-- 1 boud boud 258350 Jun  1 15:58 paper.pdf
-rw-r--r-- 1 boud boud  11609 Jun  1 15:58 paper.tex

The point is that GNU tar automatically detects whether
a file is gzipped|lzipped|whatever-zipped when x-tracting.
When c-reating, it needs to know what type of compression
you want. That's my experience, anyway.

My guess: using lzip and tar separately is probably easier
and more likely to work on more systems rather than assuming
that tar already knows about automatic lzip recognition/usage.

The alternative would be to set tar as a prerequisite
for untarring any lzipped files and then make sure we
use the installed GNU tar. My 'maneage' systems seem to
have compiled 'tar' just before 'make', so in principle
this should be doable, it seems to me.

Boud Roukema <boud>
Project Member
Mon 01 Jun 2020 09:39:40 AM UTC, comment #2: 

It is strange that it worked in your system. In my system it crashed because it tried to use `tar' on `.lz' files. When installing `lzip' it is fine because it is not compressed (`lzip.tar'). But then, when it goes to next program and try to use `tar' with `make.tar.lz' it crashed, because it is lziped!

Here I paste a couple of modifications I did in order to correct this problem, I could commit them but I prefer to post here and let you know.

diff --git a/reproduce/software/shell/pre-make-build.sh b/reproduce/software/shell/pre-make-build.sh
index 25d7d6b..5fdb7cc 100755
--- a/reproduce/software/shell/pre-make-build.sh
+++ b/reproduce/software/shell/pre-make-build.sh
@@ -123,7 +123,14 @@ download_tarball() {
build_program() {
   if ! [ -f $ibidir/$progname ]; then
     cd $tmpblddir
-    tar -xf $tardir/$tarball
+    if [ x$progname = xlzip ]; then
+      tar -xf $tardir/$tarball
+    else
+      tarball_tar=$(echo $tarball | sed -e's/.lz//')
+      lzip -k -f -d $tardir/$tarball
+      mv $tardir/$tarball_tar $tmpblddir/
+      tar -xf $tmpblddir/$tarball_tar
+    fi
     unpackdir="$progname"-"$version"
     cd $unpackdir
     ./configure --prefix=$instdir
@@ -131,6 +138,7 @@ build_program() {
     make install
     cd $topdir
     rm -rf "$tmpblddir"/$unpackdir
+    rm -rf $tmpblddir/$tarball_tar
     echo "$progname_tex $version" > $ibidir/$progname
   fi
}

Basically, it is necessary to use `lzip' with `.lz' tarballs (-k for keeping the original tarball, -f for force to overwrite and -d to decompress). But I am not able to specify a proper output (with the `-o' option), and because of that, I moved the decompressed tarball. Then use it to install the program.

This fixes that problem. But then, it crashes complaining about:

Downloading http://akhlaghi.org/src/gzip-1.10.tar.lz
./reproduce/analysis/bash/download-multi-try: full input URL (third argument) not given.

Raul Infante-Sainz <infantesainz>
Project Member
Mon 01 Jun 2020 05:58:52 AM UTC, comment #1: 

This task is done and tested on my system, its currently on the 'pre-make' branch of the development repository of maneage at https://gitlab.com/maneage/project-dev .

It would be great if you could test it on your systems by the end of today so we can hopefully merge it in as soon as possible. In particular, it should show some good improvements on macOS, where 'basic.mk' will now be built in parallel :-).

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Sun 31 May 2020 04:24:03 AM UTC, original submission:  

Make the basis of Maneage and currently we use the hosts's Make implementation for 'basic.mk' which builds make pretty early on: just after installing the compressing libraries. This has the following problems:

  • On macOS, for some reason, the parallel building of its default Make fails (we have seen this in more than one macOS). So macOS users have to build their full 'basic.mk' on a single thread! Which is very slow :-(.
  • As described in bug #58465, on FreeBSD, the host 'make' doesn't recognize the conditionals we have put.

Fortunately GNU Make is a relatively small program itself, and its creators have even included a 'build.sh' script to use to build itself (instead of calling 'make' to build 'make'!) So the system can even not have Make at all!

I am thus proposing to build GNU Make in 'configure.sh', just like the way we currently build 'flock' now and use our own GNU Make even for 'basic.mk' on any host. Doing this will make 'basic.mk' much more similar to 'high-level.mk' or 'python.mk' (allowing us to use GNU Make conventions!), while fixing the issues above.

Fortunately Make is also released with Lzip compression and Lzip is arguably the most elegant simple little program I have seen (uncompressed it is only 436 KB)! So we can also install Lzip before Make in 'configure.sh'.

We can then install 'flock' after Lzip and Make in 'configure.sh'. For Flock, we can make an Lzip tarball and download/use that instead of its standard Gzip-based tarball.

Finally, in 'basic.mk', we can also make an Lzip-based tarball for Gzip and use that to build Gzip before any package that has a 'tar.gz' tarball.

Generally, 'tar.lz' has a much higher compression ratio for source code (almost double!), so it may not be bad to later only use 'tar.lz' for our source-code archive. But this is just a brainstorm now!

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.

 

Attached Files
file #49200:  pre-make-1.tar.gz added by infantesainz (338KiB - application/x-gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by infantesainz (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-06-02 makhlaghi StatusReady For Test => Done
        Percent Complete90% => 100%
        Open/ClosedOpen => Closed
    2020-06-02 infantesainz Attached File- => Added pre-make-1.tar.gz, #49200
    2020-06-01 makhlaghi StatusIn Progress => Ready For Test
    2020-06-01 makhlaghi Percent Complete0% => 90%
    2020-05-31 makhlaghi SummaryInstall Gzip and Make before calling basic.mk => Install Lzip and Make before calling basic.mk

    Back to the top


    Powered by Savane 3.5