I added 'xcrun' to the low-level-links step and finally rebased this LONG commit (-2193 and +2468 lines!) over the 'maneage' branch as Commit c151eddbcc5f.

So I am closing this task now :-D. I'll comment/cose all the related tasks soon too.

Raul, later, if there is still a problem with 'rpcsvc-proto' on macOS, can you please define a bug?

Yes, I mean that at the end I got '.local/version-info/proglib/atlas'. So there is no problem at all.

About the Boost crash. I was able to fix it with the update of the version and some minor modifications. Have a look: https://gitlab.com/infantesainz/reproduce-raulfork/-/commit/dd62d24e

Regarding the 'rpcsvc-proto' problem, I added that line but now I obtain another error. In total I added:

if [ x$(on_mac_os) = xyes ]; then
  export CC=clang
  export CXX=clang++
  export LDFLAGS="-lintl $$LDFLAGS"
fi

And now I obtain:

checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating rpcgen/Makefile
config.status: creating rpcsvc/Makefile
config.status: creating config.h
config.status: executing depfiles commands
make[1]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
make[1]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make  all-recursive
make[2]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
Making all in rpcgen
make[3]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4/rpcgen'
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_clntout.o -MD -MP -MF .deps/rpc_clntout.Tpo -c -o rpc_clntout.o rpc_clntout.c
mv -f .deps/rpc_clntout.Tpo .deps/rpc_clntout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_cout.o -MD -MP -MF .deps/rpc_cout.Tpo -c -o rpc_cout.o rpc_cout.c
mv -f .deps/rpc_cout.Tpo .deps/rpc_cout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_hout.o -MD -MP -MF .deps/rpc_hout.Tpo -c -o rpc_hout.o rpc_hout.c
mv -f .deps/rpc_hout.Tpo .deps/rpc_hout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_main.o -MD -MP -MF .deps/rpc_main.Tpo -c -o rpc_main.o rpc_main.c
rpc_main.c:336:7: warning: 'stat64' is deprecated: first deprecated in macOS 10.6 [-Wdeprecated-declarations]
  if (stat64 (CPP, &buf) == 0)
      ^
/usr/include/sys/stat.h:402:5: note: 'stat64' has been explicitly marked deprecated here
int     stat64(const char *, struct stat64 *) __OSX_AVAILABLE_BUT_DEPRECATED(__MAC_10_5,__MAC_10_6,__IPHONE_NA,__IPHONE_NA);
        ^
rpc_main.c:1125:9: warning: 'stat64' is deprecated: first deprecated in macOS 10.6 [-Wdeprecated-declarations]
    if (stat64 (infile, &buf) < 0)
        ^
/usr/include/sys/stat.h:402:5: note: 'stat64' has been explicitly marked deprecated here
int     stat64(const char *, struct stat64 *) __OSX_AVAILABLE_BUT_DEPRECATED(__MAC_10_5,__MAC_10_6,__IPHONE_NA,__IPHONE_NA);
        ^
rpc_main.c:1132:11: warning: 'stat64' is deprecated: first deprecated in macOS 10.6 [-Wdeprecated-declarations]
      if (stat64 (outfile, &buf) < 0)
          ^
/usr/include/sys/stat.h:402:5: note: 'stat64' has been explicitly marked deprecated here
int     stat64(const char *, struct stat64 *) __OSX_AVAILABLE_BUT_DEPRECATED(__MAC_10_5,__MAC_10_6,__IPHONE_NA,__IPHONE_NA);
        ^
3 warnings generated.
mv -f .deps/rpc_main.Tpo .deps/rpc_main.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_parse.o -MD -MP -MF .deps/rpc_parse.Tpo -c -o rpc_parse.o rpc_parse.c
mv -f .deps/rpc_parse.Tpo .deps/rpc_parse.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_sample.o -MD -MP -MF .deps/rpc_sample.Tpo -c -o rpc_sample.o rpc_sample.c
mv -f .deps/rpc_sample.Tpo .deps/rpc_sample.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_scan.o -MD -MP -MF .deps/rpc_scan.Tpo -c -o rpc_scan.o rpc_scan.c
mv -f .deps/rpc_scan.Tpo .deps/rpc_scan.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_svcout.o -MD -MP -MF .deps/rpc_svcout.Tpo -c -o rpc_svcout.o rpc_svcout.c
mv -f .deps/rpc_svcout.Tpo .deps/rpc_svcout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_tblout.o -MD -MP -MF .deps/rpc_tblout.Tpo -c -o rpc_tblout.o rpc_tblout.c
mv -f .deps/rpc_tblout.Tpo .deps/rpc_tblout.Po
clang -DHAVE_CONFIG_H -I. -I..   -I/BDIR/software/installed/include  -g -O2 -MT rpc_util.o -MD -MP -MF .deps/rpc_util.Tpo -c -o rpc_util.o rpc_util.c
mv -f .deps/rpc_util.Tpo .deps/rpc_util.Po
clang  -g -O2  -lintl -o rpcgen rpc_clntout.o rpc_cout.o rpc_hout.o rpc_main.o rpc_parse.o rpc_sample.o rpc_scan.o rpc_svcout.o rpc_tblout.o rpc_util.o
make[3]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4/rpcgen'
Making all in rpcsvc
make[3]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4/rpcsvc'
../rpcgen/rpcgen -h -o klm_prot.h klm_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: klm_prot.h] Error 1
../rpcgen/rpcgen -h -o nlm_prot.h nlm_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: nlm_prot.h] Error 1
../rpcgen/rpcgen -h -o rstat.h rstat.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: rstat.h] Error 1
../rpcgen/rpcgen -h -o spray.h spray.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: spray.h] Error 1
../rpcgen/rpcgen -h -o bootparam_prot.h bootparam_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: bootparam_prot.h] Error 1
../rpcgen/rpcgen -h -o mount.h mount.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: mount.h] Error 1
../rpcgen/rpcgen -h -o rex.h rex.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: rex.h] Error 1
../rpcgen/rpcgen -h -o rusers.h rusers.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: rusers.h] Error 1
../rpcgen/rpcgen -h -o key_prot.h key_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: key_prot.h] Error 1
../rpcgen/rpcgen -h -o nfs_prot.h nfs_prot.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: nfs_prot.h] Error 1
../rpcgen/rpcgen -h -o rquota.h rquota.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: rquota.h] Error 1
../rpcgen/rpcgen -h -o sm_inter.h sm_inter.x
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cpp: line 88: xcrun: command not found
../rpcgen/rpcgen: C preprocessor failed with exit code 127
make[3]: *** [Makefile:534: sm_inter.h] Error 1
make[3]: Target 'all' not remade because of errors.
make[3]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4/rpcsvc'
make[3]: Entering directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make[3]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make[2]: *** [Makefile:347: all-recursive] Error 1
make[2]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make[1]: *** [Makefile:287: all] Error 2
make[1]: Leaving directory '/BDIR/software/build-tmp/rpcsvc-proto-1.4'
make: *** [reproduce/software/make/high-level.mk:647: /BDIR/software/installed/version-info/proglib/rpcsvc-proto-1.4] Error 2
make: Target 'all' not remade because of errors.

I think we are near of fixing also this problem. So, I will try to investigate and if I don't succeed then I will define this as bug. Thanks a lot for your help! :-)

Thanks a lot Mohammad, really impressive all changes you did in order to improve Maneage! I tested it with a clean build and I have some things to point out:

1. In the patch of M4 for macOS systems, there was a missing backslash. I guess you didn't notice it because with your system it didn't go into that `if'. Here you have the fix: https://gitlab.com/infantesainz/reproduce-raulfork/-/commit/e82b6448

2. ATLAS crashed after it was installed. This is really strange and it has to be with something related with the libraries. Attached you have the final part of the standard output. I didn't go into this in detail because as far as I remember, ATLAS is no longer necessary by any of the software of Maneage. Maybe this is because ATLAS depends on something but we did not put any prerequisite? I will check it when I have some time. I would appreciate any help if you have ;-)

3. After the previous crash of ATLAS, and because of it actually got installed, I configured again the project. This time it crashed when installing several programs, despite of I run configure with the option `-j1'. Attached there is the log.

I will try to investigate all of them in more detail. But I wanted to let you know what problems I am having right now ;-)



(file #49373)

This is a comment for checking.

Here are the Debian wiki and lintian pages on RPATH, which advise against using RPATH in Debian:

https://wiki.debian.org/RpathIssue

https://lintian.debian.org/tags/binary-or-shlib-defines-rpath.html

My guess is that this will only cause a problem in 'maneage' if a small number of libraries are updated and packages that depend on them do not have explicit dependency rules in what used to be high-level.mk or basic.mk (I haven't checked for the recent changes). I've also noticed that we tend to prefer static compilation of libraries when possible.

The Debian context is different to 'maneage', so maybe the policy here is OK.

I implemented a first draft of this task in 'basic.mk', you can see it in Commit 97d464c206. The build successfully went all the way to the end of 'basic.mk' on my system.

As described in the commit message, software management is now SO MUCH more easier! The tarballs are downloaded in the same recipe that the program is built. So to edit build instructions of a program, we only have to look into one recipe. Not like before that we had to check the tarball downloading recipe (which was long and ugly) and later, the actual build recipe.

Boud, you will also see the newly added 'reproduce/software/config/urls.conf' file. All the old software-specific URLs have been moved here, but they are all commented. If a user prefers to use them, they just have to un-comment the proper line. When the respective Make-variable is not empty, the given URL is preferred, before going onto Zenodo.

I will go onto the high-level.mk and python.mk files on Friday, but please have a look and share your thoughts if anything comes up ;-).

comment #7:

> About those two URLs, try appending them with one of the tarball names and you'll see they work ;-). They just don't have a top webpage.

OK - both work with at least one specific tarball. :)

> This proposal will change the current situation, its not just about the backup servers.
>
> Besides the backup servers, I am proposing to remove the original URLs,

So the proposal is to remove the original URLs. We'll converge
on understanding each other sooner or later. :)

> they make the code really hard to maintain, and have peculiarities. For example when I submitted Maneage to GNU last year, Richard Stallman complained that some of the links use Javascript. It wasn't easy to convince him that we also have backup servers! Because the preferred URLs were those of the software! It is still under review in GNU!

Are these URLs that require javascript or try to use javascript? I don't see any sign of a javascript engine in
.build/software/tarballs/wget-1.20.3.tar.lz . But getting maneage
to become an official GNU package would be great! (Though it
would require people who are not yet officially GNU confirmed
to do so, to protect the FSF legally.) Or is this just for FSF
status as free software? Anyway, these are interesting questions
but side topics to the main issue here.

> As an alternative, I am thinking of this: we add an optional argument for each software where the user can give a custom, software specific URL instead of one general URL. If the argument isn't set, we'll use our own backup servers.

How about something simpler: create a file
reproduce/software/config/source-urls.conf
with content something like

#bash-url-base = http://akhlaghi.org/maneage-software
bash-url-suffix = .tar.lz
binutils-url-base = http://ftp.gnu.org/gnu/binutils
binutils-url-suffix = .tar.lz
#bzip2-url = http://akhlaghi.org/maneage-software
bzip2-url-suffix = .tar.gz
...

where an absent (or commented out) ${package}-url-base or ${package}-url-suffix falls
back to a default value from reproduce/software/config/servers-backup.conf .

As before, a user-defined tarball directory would also override the URLs here; and multiple backup servers such as maneage.org and zenodo would be fallbacks.

An option should also be given to ignore *-url-base from source-urls.conf
and only use *-url-suffix .

A systematic way of handling "non-standard" URLs - such as
bitbucket non-tagged URLs "master.tar.gz" that do not have
the commit number nor version number - would also be
convenient.

I agree that this appears to be a misunderstanding.

Now it seems that the proposal would not change the present
situation: the preferred download
source will continue to be either the upstream URLs
or a local tarball directory, as chosen (or left unmodified) by the user
in reproduce/software/make/high-level.mk

The proposal is about how to handle backup (downstream) servers
of tarballs.

Is this right?

If yes, then my points are clarified - thanks. :)

Regarding the specific servers presently listed,
it seems a bit inefficient to me to use git for tarballs,
but Debian does include tarballs in specific git branches:
https://salsa.debian.org/debian-astro-team/inhomog/-/tree/pristine-tar
so it's probably a reasonable solution chosen after wide
discussion among many Debian people who are quite serious
about efficiency and sustainability.

Probably you know this, but in maneage commit 0b49fb0 I see three
backup servers, and 2 out of 3 are currently inoperative:

     12 http://gitlab.com/maneage/tarballs-software/-/raw/master

which in a browser gives me a 404 apache return code:

Page Not Found

Make sure the address is correct and the page hasn't moved.

Please contact your GitLab administrator if you think this is a mistake.

and

http://akhlaghi.org/maneage-software/

which in a browser gives me a 403 apache return code:

403 Forbidden
nginx/1.14.2

:P

Removing the original software URLs sounds like a bad idea to me, especially during the development stage.

During the development stage of a project, it would be preferable that researchers contribute their changes upstream to each of the individual software packages - contributing back to the community. If the developers of a project are slow to react, it will generally be better to create a fork to be used while waiting for the developers to either merge a pull request or make a better bug fix/improvement. This can be done reasonably efficiently with git, but requiring an extra Zenodo archiving step each time there is an update risks creating one more task that is quite slow - since AFAIK it requires using a web interface and entering metadata. If there were a convenient command line script to do this through an API, that might make sense - I don't have much Zenodo experience to be sure about this.

During the finalisation stage, having Zenodo + maneage as effectively two tarball servers is reasonable, but keeping the software source URLs (not `webpages') in the maneage system seems best to me:

(i) Since maneage does not have a full software quality assurance pipeline like in Debian, it's best that users can go directly to the source URLs, where quality control (bugs, tasks) are generally handled. The reputations of the URLs give a qualitatitve judgment of the software reliability (and software-managed trust levels can be developed too).

(ii) Zenodo is not guaranteed to always be reliable - having a single point of failure (or even just two points of failure) is risky in a system.

In Debian, the `tracker` and `download` pages both have obvious links to the upstream URLs, e.g. https://tracker.debian.org/inhomog ("homepage") https://packages.debian.org/source/stable/inhomog ("External resources - homepage")

comment #2:

> * We add an always-open task here. Anyone who wants to add a new software (or different version of an existing one) can give links to its tarball so we upload it to Zenodo and our own tarball repository.

This doesn't seem scalable to me.

I read through

git diff db2bd88..7d44e4d

but I haven't tried 7d44e4d yet.

That's a good point Raul! There are some solutions to account for this (that aren't exclusive):

• Users can give the webpage(s) containing the tarball(s) in 1) the newly added 'servers-backup.conf' configuration file. 2) the to-be-added option to specify servers at run-time. All components of this to-be-added option are now implemented in the 'from-zenodo' branch ('user_backup_urls' in configure.sh, I just have to add its option to the 'project' script.

• We add an always-open task here. Anyone who wants to add a new software (or different version of an existing one) can give links to its tarball so we upload it to Zenodo and our own tarball repository.

IMPORTANT QUESTION, PLEASE READ AND SHARE YOUR THOUGHTS.

Currently, when downloading the software tarballs, Maneage defaults to using the software's own webpage, it will then use a set of backup servers (currently using Gitlab, maneage.org and akhlaghi.org in this order).

But managing all the different software URLs (as the primary download URL) is VERY ANNOYING and sometimes problematic because some use strange servers that may use some Javascript. It also makes the software downloading list very long, hard to read and prone to bugs.

The main reason we preferred the original software URL until now was that using Gitlab as a source to download a few hundred megabytes of data was not too ethical (I didn't feel comfortable occupying their traffic limits with such large downloads, thus slowing down other developers), and our own servers of maneage.org and akhlaghi.org are privately funded, and traffic plays an important role in server costs (both are currently paid by myself privately).

Recently a good alternative occurred to me: Zenodo! It is precisely defined for this type of job (space and traffic), while giving a reliable identifier (DOI). So I uploaded all our current tarballs to zenodo.3883409. This DOI will not be changed even after we add new tarballs (newer versions of existing software or new software overall): it will always point to the most recent version.

The only complication was that Zenodo doesn't have a default way of extracting the most recent version's identifier, so I had to use a hack, after consulting Zenodo developers (implemented in the from-zenodo branch).

With this feature implemented, if the download from a software's server fails, Maneage will automatically use the most recent uploaded version to Zenodo (note that we will never remove a file from Zenodo, we will always only add to it).

Now that we have a reliable server for downloading software source codes without any trouble, I am thus proposing to remove all the software-specific URLs.

What do you think?