taskManeage - Tasks: task #15686, Removing original software URLs...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15686: Removing original software URLs from Maneage?

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Wed 10 Jun 2020 11:11:12 PM UTC
   
 
Should Start On:  Tue 09 Jun 2020 11:00:00 PM UTC Should be Finished on:  Tue 09 Jun 2020 11:00:00 PM UTC
Category:  Software Priority:  5 - Normal
Status:  Done Privacy:  Public
Assigned to:  makhlaghi Percent Complete:  100%
Open/Closed:  Closed Effort:  0.00

Jump to the original submission

Sun 28 Jun 2020 03:23:39 AM UTC, comment #20: 

Thanks a lot Raul for the fast check and bug #58663.

The Bison fix is now on the core Maneage branch as Commit 1729f42b01aa1 ;-).


Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 28 Jun 2020 12:47:30 AM UTC, comment #19: 

Thanks a lot Mohammad. I got all changes and run configure step obtaining only two problems:

1. Crash installing Bison. I was able to fix this problem just updating the version to the most recent :-) Since this was causing a crash I have not consider this to be a "regular software update". So, you can see the commit here: https://gitlab.com/infantesainz/reproduce-raulfork/-/commit/b079a025f0

2. Crash installing `libtirpc'. I am defining a bug for this.

The rest of software is installed without any problem! :-D

Raul Infante-Sainz <infantesainz>
Group Member
Sat 27 Jun 2020 05:39:09 PM UTC, comment #18: 

I added 'xcrun' to the low-level-links step and finally rebased this LONG commit (-2193 and +2468 lines!) over the 'maneage' branch as Commit c151eddbcc5f.

So I am closing this task now :-D. I'll comment/cose all the related tasks soon too.

Raul, later, if there is still a problem with 'rpcsvc-proto' on macOS, can you please define a bug?


Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 27 Jun 2020 12:20:15 AM UTC, comment #17: 

Thanks a lot Raul, the fix in rpcsvc-proto is now implemented in Commit d318f14a57da418 of the 'only-gnu-make' branch.

I just had a look at the error messages that followed, and noticed that all of them are essentially the same:


../rpcgen/rpcgen -h -o rquota.h rquota.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127


Essentially, it can't find 'xcrun'! So I had a look in the internet and noticed that 'xcrun' is actually a low-level tool in macOS (see its man page). So can you try this:

  • Add 'low-level-links' target 'basic.mk' (under the 'macOS specific' step).


  • Delete '.local/version-info/proglib/low-level-links'


  • Rebuild the project.


Hopefully this particular problem should be fixed afterwards, let's see if anything else comes up...

If everything work, I'll rebase this whole branch as one commit in the "Maneage" branch ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 26 Jun 2020 04:53:07 PM UTC, comment #16: 

Yes, I mean that at the end I got '.local/version-info/proglib/atlas'. So there is no problem at all.

About the Boost crash. I was able to fix it with the update of the version and some minor modifications. Have a look: https://gitlab.com/infantesainz/reproduce-raulfork/-/commit/dd62d24e

Regarding the 'rpcsvc-proto' problem, I added that line but now I obtain another error. In total I added:


if [ x$(on_mac_os) = xyes ]; then
  export CC=clang
  export CXX=clang++
  export LDFLAGS="-lintl $$LDFLAGS"
fi


And now I obtain:


checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating rpcgen/Makefile
config.status: creating rpcsvc/Makefile
config.status: creating config.h
config.status: executing depfiles commands
make[1]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
make[1]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make  all-recursive
make[2]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
Making all in rpcgen
make[3]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4/rpcgen'
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_clntout.o -MD -MP -MF .deps/rpc_clntout.Tpo -c -o rpc_clntout.o rpc_clntout.c
mv -f .deps/rpc_clntout.Tpo .deps/rpc_clntout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_cout.o -MD -MP -MF .deps/rpc_cout.Tpo -c -o rpc_cout.o rpc_cout.c
mv -f .deps/rpc_cout.Tpo .deps/rpc_cout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_hout.o -MD -MP -MF .deps/rpc_hout.Tpo -c -o rpc_hout.o rpc_hout.c
mv -f .deps/rpc_hout.Tpo .deps/rpc_hout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_main.o -MD -MP -MF .deps/rpc_main.Tpo -c -o rpc_main.o rpc_main.c
rpc_main.c:336:7: warning: 'stat64' is deprecated: first deprecated in macOS 10.6 [-Wdeprecated-declarations]
  if (stat64 (CPP, &buf) == 0)
      ^
/usr/include/sys/stat.h:402:5: note: 'stat64' has been explicitly marked deprecated here
int     stat64(const char *, struct stat64 *) __OSX_AVAILABLE_BUT_DEPRECATED(__MAC_10_5,__MAC_10_6,__IPHONE_NA,__IPHONE_NA);
        ^
rpc_main.c:1125:9: warning: 'stat64' is deprecated: first deprecated in macOS 10.6 [-Wdeprecated-declarations]
    if (stat64 (infile, &buf) < 0)
        ^
/usr/include/sys/stat.h:402:5: note: 'stat64' has been explicitly marked deprecated here
int     stat64(const char *, struct stat64 *) __OSX_AVAILABLE_BUT_DEPRECATED(__MAC_10_5,__MAC_10_6,__IPHONE_NA,__IPHONE_NA);
        ^
rpc_main.c:1132:11: warning: 'stat64' is deprecated: first deprecated in macOS 10.6 [-Wdeprecated-declarations]
      if (stat64 (outfile, &buf) < 0)
          ^
/usr/include/sys/stat.h:402:5: note: 'stat64' has been explicitly marked deprecated here
int     stat64(const char *, struct stat64 *) __OSX_AVAILABLE_BUT_DEPRECATED(__MAC_10_5,__MAC_10_6,__IPHONE_NA,__IPHONE_NA);
        ^
3 warnings generated.
mv -f .deps/rpc_main.Tpo .deps/rpc_main.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_parse.o -MD -MP -MF .deps/rpc_parse.Tpo -c -o rpc_parse.o rpc_parse.c
mv -f .deps/rpc_parse.Tpo .deps/rpc_parse.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_sample.o -MD -MP -MF .deps/rpc_sample.Tpo -c -o rpc_sample.o rpc_sample.c
mv -f .deps/rpc_sample.Tpo .deps/rpc_sample.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_scan.o -MD -MP -MF .deps/rpc_scan.Tpo -c -o rpc_scan.o rpc_scan.c
mv -f .deps/rpc_scan.Tpo .deps/rpc_scan.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_svcout.o -MD -MP -MF .deps/rpc_svcout.Tpo -c -o rpc_svcout.o rpc_svcout.c
mv -f .deps/rpc_svcout.Tpo .deps/rpc_svcout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_tblout.o -MD -MP -MF .deps/rpc_tblout.Tpo -c -o rpc_tblout.o rpc_tblout.c
mv -f .deps/rpc_tblout.Tpo .deps/rpc_tblout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_util.o -MD -MP -MF .deps/rpc_util.Tpo -c -o rpc_util.o rpc_util.c
mv -f .deps/rpc_util.Tpo .deps/rpc_util.Po
clang  -g -O2  -lintl -o rpcgen rpc_clntout.o rpc_cout.o rpc_hout.o rpc_main.o rpc_parse.o rpc_sample.o rpc_scan.o rpc_svcout.o rpc_tblout.o rpc_util.o
make[3]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4/rpcgen'
Making all in rpcsvc
make[3]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4/rpcsvc'
../rpcgen/rpcgen -h -o klm_prot.h klm_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: klm_prot.h] Error 1
../rpcgen/rpcgen -h -o nlm_prot.h nlm_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: nlm_prot.h] Error 1
../rpcgen/rpcgen -h -o rstat.h rstat.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: rstat.h] Error 1
../rpcgen/rpcgen -h -o spray.h spray.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: spray.h] Error 1
../rpcgen/rpcgen -h -o bootparam_prot.h bootparam_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: bootparam_prot.h] Error 1
../rpcgen/rpcgen -h -o mount.h mount.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: mount.h] Error 1
../rpcgen/rpcgen -h -o rex.h rex.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: rex.h] Error 1
../rpcgen/rpcgen -h -o rusers.h rusers.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: rusers.h] Error 1
../rpcgen/rpcgen -h -o key_prot.h key_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: key_prot.h] Error 1
../rpcgen/rpcgen -h -o nfs_prot.h nfs_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: nfs_prot.h] Error 1
../rpcgen/rpcgen -h -o rquota.h rquota.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: rquota.h] Error 1
../rpcgen/rpcgen -h -o sm_inter.h sm_inter.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: sm_inter.h] Error 1
make[3]: Target 'all' not remade because of errors.
make[3]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4/rpcsvc'
make[3]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make[3]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make[2]: *** [Makefile:347: all-recursive] Error 1
make[2]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make[1]: *** [Makefile:287: all] Error 2
make[1]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make: *** [reproduce/software/make/high-level.mk:647: /BDIR/software/installed/version-info/proglib/rpcsvc-proto-1.4] Error 2
make: Target 'all' not remade because of errors.


I think we are near of fixing also this problem. So, I will try to investigate and if I don't succeed then I will define this as bug. Thanks a lot for your help! :-)

Raul Infante-Sainz <infantesainz>
Group Member
Fri 26 Jun 2020 02:17:59 AM UTC, comment #15: 

Great! Thanks a lot for the checks Raul!

Thanks also for the nice catch on the incorrect removal of the backslash! I have pushed your commit over mine in that branch, so it will be rebased in the final commit later ;-).

About ATLAS, I just defined bug #58655. Let's continue this discussion there. But just one thing, by "it actually got installed", do you mean that you have '.local/version-info/proglib/atlas'? If so, then there is actually no problem! Because as I later found in that bug, all those errors have been ignored!

After ATLAS, apparently three software failed: 'boost', 'libtirpc' and 'rpcsvc-proto'.

  • I searched for the problem in Boost and found that it was a known issue in November 2019. I then noticed that the most recent version came just about two months ago. So can you please try updating the version number to 1.73.0 to see if this particular problem is already fixed? You can just uncomment the "boost-url" variable in the new 'urls.conf'.


  • About the 'libtirpc' errors, they all seem to be related to the  pthreads library.


  • About the 'rpcsvc-proto' problem, it seems to be related to Gettext's internationalization library, in particular the lines below. You should have a '.local/lib/libintl*' since we now install Gettext on macOS. So as a test, can you try adding '-lintl' to the LDFLAGS before building rpcsvc-proto? This line: 'export LDFLAGS="-lintl $$LDFLAGS"'



Undefined symbols for architecture x86_64:
  "_libintl_gettext", referenced from:
      _main in rpc_main.o
      _usage in rpc_main.o
      _checkfiles in rpc_main.o
      _c_output in rpc_main.o
      _h_output in rpc_main.o
      _l_output in rpc_main.o
      _s_output in rpc_main.o
      ...
  "_libintl_setlocale", referenced from:
      _main in rpc_main.o
  "_libintl_textdomain", referenced from:
      _main in rpc_main.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)


If these proposals don't fix the problem, can you define new bugs for these software on macOS? Since no one needs them seriously now, we can leave the bugs open until we either randomly find a solution or someone is motivated enough to look into them ;-).

Generally, to (temporarily) remove a package from '--all-highlevel', you can simply comment the "atlast-version" line in 'versions.conf'). But this just lead me to an idea: Since such known problems can be very annoying in a general check (with '--all-highlevel'), one option is to add an automatic step to remove such known problematic software (for specific OSs, citing the particular bug) from the list of packages that are installed with '--all-highlevel'.

What do you think?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 25 Jun 2020 11:21:05 PM UTC, comment #14: 

Thanks a lot Mohammad, really impressive all changes you did in order to improve Maneage! I tested it with a clean build and I have some things to point out:

1. In the patch of M4 for macOS systems, there was a missing backslash. I guess you didn't notice it because with your system it didn't go into that `if'. Here you have the fix: https://gitlab.com/infantesainz/reproduce-raulfork/-/commit/e82b6448

2. ATLAS crashed after it was installed. This is really strange and it has to be with something related with the libraries. Attached you have the final part of the standard output. I didn't go into this in detail because as far as I remember, ATLAS is no longer necessary by any of the software of Maneage. Maybe this is because ATLAS depends on something but we did not put any prerequisite? I will check it when I have some time. I would appreciate any help if you have ;-)

3. After the previous crash of ATLAS, and because of it actually got installed, I configured again the project. This time it crashed when installing several programs, despite of I run configure with the option `-j1'. Attached there is the log.

I will try to investigate all of them in more detail. But I wanted to let you know what problems I am having right now ;-)



(file #49373)

Raul Infante-Sainz <infantesainz>
Group Member
Thu 25 Jun 2020 07:12:59 AM UTC, comment #13: 

Thanks Boud, in Debian (or generally, any complete ooerating system), this is indeed the case. But Maneage is installed within another operating system, so the differing library versions between Maneage and the host can cause conflicts leading to programs crashing (in some cases even where LD_LIBRAY_PATH was set in the Makefile!)! This was arguably the most frustrating thing for Raul and me last year when we were adding the first draft of the software building rules.

Having RPATH within the executable also allows it to be run outside of Maneage's Makefiles. For example you can run '.local/bin/awk' to use your project's AWK in your host's shell for other things!

Infact when I am on a large institute computer (where I don't have root access and can't easily change the core software versions, and they are usually very old versions), I just add one of my Maneaged project's installed software directories to my PATH in my account's '.bashrc'. So I can use my modern and fully controlled versions of things like Bash, AWK or etc :-).

About the static build being preferred, this is an old relic which I guess we should remove! It is effectively disabled in "configure.sh". This was because many software unfortunately don't support static builds, and more importantly, in a static build you have no way of knowing (to my knowledge!) what has been linked! This resulted in some hard-to-debug situations in those first days!

Now, you can run this command on any Maneaged software (even outside of Maneage) in GNU/Linux OSs (in macOS it is 'otool -L') to make sure what libraries it links with:


ldd .local/bin/awk


Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 25 Jun 2020 06:07:52 AM UTC, comment #12: 

This is a comment for checking.

Here are the Debian wiki and lintian pages on RPATH, which advise against using RPATH in Debian:

https://wiki.debian.org/RpathIssue

https://lintian.debian.org/tags/binary-or-shlib-defines-rpath.html

My guess is that this will only cause a problem in 'maneage' if a small number of libraries are updated and packages that depend on them do not have explicit dependency rules in what used to be high-level.mk or basic.mk (I haven't checked for the recent changes). I've also noticed that we tend to prefer static compilation of libraries when possible.

The Debian context is different to 'maneage', so maybe the policy here is OK.

Boud Roukema <boud>
Group Member
Thu 25 Jun 2020 01:03:41 AM UTC, comment #11: 

             !!!!!!!!!!!!!!!!!!!!!!!!!!
             !!!!! VERY IMPORTANT !!!!!
             !!!!!!!!!!!!!!!!!!!!!!!!!!

Implementing this task forced me to go a little low-level, and very soon started a chain-reaction(!), in one way or another, merging into these tasks: task #15399, task #15389, task #15272, task #15359 and task #15330!

It effectively took my free time in the whole week, and resulted in an almost-complete rewrite of the software building phase of Maneage, making it SO MUCH more robust and easier to read/modify.

You can see the changes in Commit 603ba6d459e on my development branch. Just for the record, I am also putting the full commit message at the end of this post.

I already checked the building of all the software (with the new '--all-highlevel' option) in a 'slim' Debian-stable Docker image (while testing task #15389!) and it worked nicely (after almost two days of fixing issues due to the low-level change in each package!). But it would be great if you could also try it out and look into it in the next two days. If a problem isn't found or a suggestion isn't made by Saturday. I'll merge it into the core Maneage branch.

In particular, Raul, it would be great if you could try this branch on your macOS (don't forget the '--all-highlevel' option ;-)).


 IMPORTANT: Software recipes use GNU Make, many other improvements

POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause
conflicts to your project if you have changed the software building
Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and
'python.mk'). If your project has only added analysis, it shouldn't affect
it, but still read through the points for using the newly added
features. In general, things have become much more clear and easier to
read/execute now ;-).

This is a large commit, involving a long series of corrections in a
differnt branch for an initial low-level change. Other changes naturally
came up as the low-level structure was improved.

 - Until now, Maneage needed the host to have a 'make' implementation which
   was necessary to build Lzip (Lzip is then used to uncompress the source
   of GNU Make). But this is not the case in the minimalist/slim versions
   of operating systems (for example used to build Docker images). After
   consulting Antonio Diaz Diaz (creator of Lzip), he kindly added the
   necessary functionality to Lzip and Lzip can now be built without Make.
   Hence we don't need this assumption any more. With this commit, Lzip and
   GNU Make are built without Make, allowing everything else to be safely
   built with our own custom version of GNU Make and not using the host's
   'make' at all.

 - Until recently (Commit 3d8aa595) GNU Make was built in
   'basic.mk'. Therefore 'basic.mk' was written in a way that it can be
   used with other 'make' implementations also (important shell commands
   starting with '&&' and ending in '\', it also didn't allow any
   comments!). Furthermore, to help in style uniformity, the rules in
   'high-level.mk' and 'python.mk' also followed a similar structure. But
   due to the point above, we can now guarantee that GNU Make is used from
   the very first Makefile, so this hard-to-read structure has been removed
   in the software build recipes.

 - Until now, the default backup servers where at some fixed URLs, on our
   own pages or on Gitlab. But recently we uploaded all the necessary
   software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is
   more suitable for this task (it promises longevity, has a fixed DOI,
   while allowing us to add new content, or new software tarball
   versions). With this commit, a small script has been written to extract
   the most recent Zenodo upload link from the Zenodo DOI and use it for
   downloading the software source codes.

 - Until now, we primarily used the webpage of each software as the primary
   source for downloading its tarball. But this caused many problems: 1)
   Some of them needed Javascript before the download, 2) Some URLs had a
   complex dependency on the version number, 3) some servers would be
   randomly down for maintenance and etc. So thanks to the point above, we
   now use the Zenodo server as the primary download location. However, if
   a user wants to use a custom software that is not (yet!) on Zenodo, the
   download script gives priority to a custom URL that the users can give
   as Make variables. If that variable is defined, then the script will use
   that URL before going onto Zenodo. We now have a special place for such
   URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a
   good documentation themselvs) are preserved here, but are commented by
   default.

 - The downloading and checksum verification step has been moved into a
   Make function called 'import-source' (defined in the 'build-rules.mk'
   and loaded in all software Makefiles). Having taken all the low-level
   steps there, I noticed that there is no more need for having the tarball
   as a separate target! So with this commit, a single rule is the only
   place that needs to be edited/added (greatly simplifying the software
   building Makefiles).

 - Following task #15272, A new option has been added to the './project'
   script called '--all-highlevel'. When this option is given, the contents
   of 'TARGETS.conf' are ignored and a script goes through all the software
   (by parsing the 'versions.conf' file) and sets all high-level software
   as targets to be built. I added and used this new option to confirm the
   extensive changes made in all the software building recipes!

 - Many of the software hadn't been tested for a long time! So after using
   the newly added '--all-highlevel', I found that some need to be
   updated. In general, with this commit 'libpaper' and 'pcre' were added
   as totally new software, and the versions of the following software was
   updated: 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in'
   shell script was added in 'reproduce/software/shell/' which is installed
   with 'libpaper'.

 - Even though we intentionally add the necessary flags to add RPATH inside
   the built executable at compilation time, some software don't do it
   (different software on different operating systems!). Until now, we were
   simply checking if the operating system is macOS, but a more robust
   check is to see if 'patchelf' (used for manually updating RPATH in
   GNU/Linux systems and not installed on macOS) is present or not. Because
   of this, 'patchelf' has been put as a top-level prerequisite, right
   after Tar and is installed before anything else.

 - In 'versions.conf', GNU Libtool is recognized as 'libtool', but in
   'basic.mk', it was 'glibtool'! This caused many confusions and is
   corrected with this commit (in 'basic.mk', it is also 'libtool').

 - A new argument is added to the './project' script to allow easy loading
   of the project's shell and environment for fast/temporary testing of
   things in the same environment as the project. Before activating the
   project's shell, we completely remove all host environment variables to
   simulate the project's environment. It can be called with this command:
   './project shell'. Note that it doesn't yet have any prompt! all lines
   just start with a '$'. We should add one later.



Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 19 Jun 2020 04:29:37 AM UTC, comment #10: 

I implemented a first draft of this task in 'basic.mk', you can see it in Commit 97d464c206. The build successfully went all the way to the end of 'basic.mk' on my system.

As described in the commit message, software management is now SO MUCH more easier! The tarballs are downloaded in the same recipe that the program is built. So to edit build instructions of a program, we only have to look into one recipe. Not like before that we had to check the tarball downloading recipe (which was long and ugly) and later, the actual build recipe.

Boud, you will also see the newly added 'reproduce/software/config/urls.conf' file. All the old software-specific URLs have been moved here, but they are all commented. If a user prefers to use them, they just have to un-comment the proper line. When the respective Make-variable is not empty, the given URL is preferred, before going onto Zenodo.

I will go onto the high-level.mk and python.mk files on Friday, but please have a look and share your thoughts if anything comes up ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 17 Jun 2020 03:42:16 AM UTC, comment #9: 

Yes, I also had a similar thing in mind. Besides possible technical usage, the proper complete URL of a software has a lot of value as documentation.

I am a little busy right now, but hopefully I'll implement it shortly.

About the Javascript, it did happen before for software that were on servers like SourceForge. If I remember correctly, such servers require the downloading software (for example wget or curl) to do some potentially-non-ethical operations on the host before they release the actual tarball.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 16 Jun 2020 06:20:39 AM UTC, comment #8: 

comment #7:

> About those two URLs, try appending them with one of the tarball names and you'll see they work ;-). They just don't have a top webpage.


OK - both work with at least one specific tarball. :)

> This proposal will change the current situation, its not just about the backup servers.
>
> Besides the backup servers, I am proposing to remove the original URLs,


So the proposal is to remove the original URLs. We'll converge
on understanding each other sooner or later. :)

> they make the code really hard to maintain, and have peculiarities. For example when I submitted Maneage to GNU last year, Richard Stallman complained that some of the links use Javascript. It wasn't easy to convince him that we also have backup servers! Because the preferred URLs were those of the software! It is still under review in GNU!


Are these URLs that require javascript or try to use javascript? I don't see any sign of a javascript engine in
.build/software/tarballs/wget-1.20.3.tar.lz . But getting maneage
to become an official GNU package would be great! (Though it
would require people who are not yet officially GNU confirmed
to do so, to protect the FSF legally.) Or is this just for FSF
status as free software? Anyway, these are interesting questions
but side topics to the main issue here.

> As an alternative, I am thinking of this: we add an optional argument for each software where the user can give a custom, software specific URL instead of one general URL. If the argument isn't set, we'll use our own backup servers.


How about something simpler: create a file
reproduce/software/config/source-urls.conf
with content something like

#bash-url-base = http://akhlaghi.org/maneage-software
bash-url-suffix = .tar.lz
binutils-url-base = http://ftp.gnu.org/gnu/binutils
binutils-url-suffix = .tar.lz
#bzip2-url = http://akhlaghi.org/maneage-software
bzip2-url-suffix = .tar.gz
...

where an absent (or commented out) ${package}-url-base or ${package}-url-suffix falls
back to a default value from reproduce/software/config/servers-backup.conf .

As before, a user-defined tarball directory would also override the URLs here; and multiple backup servers such as maneage.org and zenodo would be fallbacks.

An option should also be given to ignore *-url-base from source-urls.conf
and only use *-url-suffix .

A systematic way of handling "non-standard" URLs - such as
bitbucket non-tagged URLs "master.tar.gz" that do not have
the commit number nor version number - would also be
convenient.

Boud Roukema <boud>
Group Member
Tue 16 Jun 2020 12:03:49 AM UTC, comment #7: 

About those two URLs, try appending them with one of the tarball names and you'll see they work ;-). They just don't have a top webpage.

This proposal will change the current situation, its not just about the backup servers.

Besides the backup servers, I am proposing to remove the original URLs, they make the code really hard to maintain, and have peculiarities. For example when I submitted Maneage to GNU last year, Richard Stallman complained that some of the links use Javascript. It wasn't easy to convince him that we also have backup servers! Because the preferred URLs were those of the software! It is still under review in GNU!

As an alternative, I am thinking of this: we add an optional argument for each software where the user can give a custom, software specific URL instead of one general URL. If the argument isn't set, we'll use our own backup servers.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 15 Jun 2020 10:20:47 PM UTC, comment #6: 

I agree that this appears to be a misunderstanding.

Now it seems that the proposal would not change the present
situation: the preferred download
source will continue to be either the upstream URLs
or a local tarball directory, as chosen (or left unmodified) by the user
in reproduce/software/make/high-level.mk

The proposal is about how to handle backup (downstream) servers
of tarballs.

Is this right?

If yes, then my points are clarified - thanks. :)

Regarding the specific servers presently listed,
it seems a bit inefficient to me to use git for tarballs,
but Debian does include tarballs in specific git branches:
https://salsa.debian.org/debian-astro-team/inhomog/-/tree/pristine-tar
so it's probably a reasonable solution chosen after wide
discussion among many Debian people who are quite serious
about efficiency and sustainability.

Probably you know this, but in maneage commit 0b49fb0 I see three
backup servers, and 2 out of 3 are currently inoperative:


     12 http://gitlab.com/maneage/tarballs-software/-/raw/master

which in a browser gives me a 404 apache return code:

Page Not Found

Make sure the address is correct and the page hasn't moved.

Please contact your GitLab administrator if you think this is a mistake.


and

http://akhlaghi.org/maneage-software/

which in a browser gives me a 403 apache return code:

403 Forbidden
nginx/1.14.2


:P

Boud Roukema <boud>
Group Member
Mon 15 Jun 2020 02:57:52 AM UTC, comment #5: 

Thanks for your thoughts Boud,

Maybe my initial wording of the proposed solution was not too good, so let me clarify the suggestion:

In the current Maneage framework, there are a set of backup servers and recently Zenodo has been added is well.

These backup servers currently act like this: when the original software URL doesn't respond (for any reason), Maneage will start looking for it in these backup servers one-by-one (first in Zenodo, then in the servers of the configuration file).

Now, suppose you are using some software packages that are not yet in any of the official Maneage backup servers. You can put them in your own server, and add this server to the configuration file.

After time you run './project configure', it also look into your desired server and ultimately find the file (depending on the place you put your new server).

Since this file is under version control, if I later want to reproduce your project, when I run './project configure', it will also look into your server for the tarball and bring it for me.

So you see, you don't have to add your software's tarball to the official Maneage Zenodo page, or the official Maneage software repository to use your custom software.

Of course, before publication it would be good if you submit the build-instructions of your newly added software as a merge-request for the core Maneage branch so others can benefit from it (most importantly yourself, in your next project). In the meantime, you can also submit their tarballs to be included into the Maneage archival system (the Git repo and Zenodo).

But until that is done, your project can continue and you don't have to wait ;-).

About Zenodo being a single point of failure: fortunately that is  not the case, currently we have the Maneage-archived software tarballs in three other servers, as you can see from backup servers.

Zenodo is just one other redundancy, the currently promises to be the most reliable, and free solution, designed for transferring large files.

Does this clarify the points you raised?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 12 Jun 2020 10:54:48 PM UTC, comment #4: 

Removing the original software URLs sounds like a bad idea to me, especially during the development stage.

During the development stage of a project, it would be preferable that researchers contribute their changes upstream to each of the individual software packages - contributing back to the community. If the developers of a project are slow to react, it will generally be better to create a fork to be used while waiting for the developers to either merge a pull request or make a better bug fix/improvement. This can be done reasonably efficiently with git, but requiring an extra Zenodo archiving step each time there is an update risks creating one more task that is quite slow - since AFAIK it requires using a web interface and entering metadata. If there were a convenient command line script to do this through an API, that might make sense - I don't have much Zenodo experience to be sure about this.

During the finalisation stage, having Zenodo + maneage as effectively two tarball servers is reasonable, but keeping the software source URLs (not `webpages') in the maneage system seems best to me:

(i) Since maneage does not have a full software quality assurance pipeline like in Debian, it's best that users can go directly to the source URLs, where quality control (bugs, tasks) are generally handled. The reputations of the URLs give a qualitatitve judgment of the software reliability (and software-managed trust levels can be developed too).

(ii) Zenodo is not guaranteed to always be reliable - having a single point of failure (or even just two points of failure) is risky in a system.

In Debian, the `tracker` and `download` pages both have obvious links to the upstream URLs, e.g. https://tracker.debian.org/inhomog ("homepage") https://packages.debian.org/source/stable/inhomog ("External resources - homepage")

comment #2:

> * We add an always-open task here. Anyone who wants to add a new software (or different version of an existing one) can give links to its tarball so we upload it to Zenodo and our own tarball repository.


This doesn't seem scalable to me.

I read through

git diff db2bd88..7d44e4d

but I haven't tried 7d44e4d yet.

Boud Roukema <boud>
Group Member
Thu 11 Jun 2020 02:03:26 PM UTC, comment #3: 

We have the solution then! It is up to the user to use the "Official Maneage repository" or any other he/she wants :-)
In such scenario it is not longer necessary to keep the individual URLs and things will be much more easy and clear.

Raul Infante-Sainz <infantesainz>
Group Member
Thu 11 Jun 2020 01:59:03 PM UTC, comment #2: 

That's a good point Raul! There are some solutions to account for this (that aren't exclusive):

  • Users can give the webpage(s) containing the tarball(s) in 1) the newly added 'servers-backup.conf' configuration file. 2) the to-be-added option to specify servers at run-time. All components of this to-be-added option are now implemented in the 'from-zenodo' branch ('user_backup_urls' in configure.sh, I just have to add its option to the 'project' script.


  • We add an always-open task here. Anyone who wants to add a new software (or different version of an existing one) can give links to its tarball so we upload it to Zenodo and our own tarball repository.
Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 11 Jun 2020 09:46:50 AM UTC, comment #1: 

I think it is a really good idea to do that because it will simplify the way the software is managed. However, I see some problems.

It will not be up to the user the election of a particular version of a given software, right? If we remove all software-specific URL (I understand by this to remove each individual URL to each program in `basic.mk', `high-level.mk' and `python.mk'), then the version of each program is only the version we have uploaded to zenodo. Is this correct?

In principle, I think the advantages of keep things easier are far better than possible annoying of a particular person needing a particular version of a program. And even in this case, we can always upload the needed version of the program. Also, the fact that we will never remove any tarball ensures that we are safe in this sense.

Raul Infante-Sainz <infantesainz>
Group Member
Wed 10 Jun 2020 11:11:12 PM UTC, original submission:  

IMPORTANT QUESTION, PLEASE READ AND SHARE YOUR THOUGHTS.

Currently, when downloading the software tarballs, Maneage defaults to using the software's own webpage, it will then use a set of backup servers (currently using Gitlab, maneage.org and akhlaghi.org in this order).

But managing all the different software URLs (as the primary download URL) is VERY ANNOYING and sometimes problematic because some use strange servers that may use some Javascript. It also makes the software downloading list very long, hard to read and prone to bugs.

The main reason we preferred the original software URL until now was that using Gitlab as a source to download a few hundred megabytes of data was not too ethical (I didn't feel comfortable occupying their traffic limits with such large downloads, thus slowing down other developers), and our own servers of maneage.org and akhlaghi.org are privately funded, and traffic plays an important role in server costs (both are currently paid by myself privately).

Recently a good alternative occurred to me: Zenodo! It is precisely defined for this type of job (space and traffic), while giving a reliable identifier (DOI). So I uploaded all our current tarballs to zenodo.3883409. This DOI will not be changed even after we add new tarballs (newer versions of existing software or new software overall): it will always point to the most recent version.

The only complication was that Zenodo doesn't have a default way of extracting the most recent version's identifier, so I had to use a hack, after consulting Zenodo developers (implemented in the from-zenodo branch).

With this feature implemented, if the download from a software's server fails, Maneage will automatically use the most recent uploaded version to Zenodo (note that we will never remove a file from Zenodo, we will always only add to it).

Now that we have a reliable server for downloading software source codes without any trouble, I am thus proposing to remove all the software-specific URLs.

What do you think?

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #49373:  logs.tar.gz added by infantesainz (264KiB - application/x-gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by infantesainz (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-06-27 makhlaghi StatusReady For Test Done
        Percent Complete90% 100%
        Open/ClosedOpen Closed
    2020-06-25 infantesainz Attached File- Added logs.tar.gz, #49373
    2020-06-25 makhlaghi StatusIn Progress Ready For Test
    2020-06-19 makhlaghi Percent Complete60% 90%
        Assigned toNone makhlaghi

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code