taskReproducible paper template - Tasks: task #15682, maneage-lint: check for host...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15682: maneage-lint: check for host system shebang lines in executable scripts

Submitted by:  Boud Roukema <boud>
Submitted on:  Tue 09 Jun 2020 01:41:57 AM UTC  
 
Should Start On:  Tue 09 Jun 2020 12:00:00 AM UTC Should be Finished on:  Tue 09 Jun 2020 12:00:00 AM UTC
Category:  None Priority:  5 - Normal
Status:  None Privacy:  Public
Percent Complete:  0% Assigned to:  None
Open/Closed:  Open Effort:  0.00

( Jump to the original submission)

Sun 21 Jun 2020 06:03:54 PM UTC, comment #6: 

Just a comment on the wider question of lint type checkers in general: A very real-world example is the Debian lintian package
https://tracker.debian.org/pkg/lintian

This gives us a real-life warning of the potential complexity of the overall task. Right now, the lintian tracker URL points to 343 open bugs in lintian itself, of which 326 do not yet have patches ready. This is out of a total of about 2700 bugs (closed and unclosed) altogether.

In other words, the tool for checking that a software package satisfies the required Debian standards is itself rather buggy. My impression from using this in practice is that that tends to mean that lintian warns about what it thinks are bugs, but the documentation is insufficiently clear in explaining what is considered non-standard or unsafe. Sometimes lintian has false positives - it warns about a problem that is not really a problem; the particular check involved is insufficiently intelligent to understand all cases. So someone using lintian has to him/herself be sufficiently skilled (and able to learn more) to decide how to handle the warnings. And preferably s/he should also file a bug report and/or propose a fix.

Boud Roukema <boud>
Project Member
Tue 09 Jun 2020 09:57:11 PM UTC, comment #5: 

Great! Thanks for the check, this is just what I meant ;-)!

So this seems to be a good solution. We should just change the directory part of the shebang to '/usr/bin/env '

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Tue 09 Jun 2020 09:21:49 PM UTC, comment #4: 

comment #3:

> The reason is that within the project's Make (as called in the top-level 'project' script) we completely close-off the host's environment and have our own PATH. So 'env PROGNAME' should expand to the PROGNAME in Maneage, right? Even if 'env' doesn't belong to Maneage itself.


I don't think that expand is the right word; the effect is that the environment variables from the environment are used. When running a make rule from inside our .mk files, we have PATH=.local/bin . So the behaviour should be something like this:

$ cat .local/bin/dummy
#!/usr/bin/env bash
printf "${BASH_VERSION}\n\n"

$ PATH=.local/bin dummy
5.0.11(1)-release

$ .local/bin/dummy
4.4.12(1)-release

Boud Roukema <boud>
Project Member
Tue 09 Jun 2020 06:34:30 PM UTC, comment #3: 

Very nice point!

Using '.local/bin/<executable>' is one solution, but as you said it will potentially cause problems when users change directory within a recipe (it happens, for example when building the LaTeX paper in 'paper.mk'),

I have sometimes been forced to do this in my research because some research software only write their outputs in the current directory, so the only way to avoid writing in the source directory is to 'cd' to the special directory, run it there, and come back up to the source!

Maybe using the host's 'env' is the most portable solution (because if we want to use Maneage's 'env', we'll have the same length-problem!).

The reason is that within the project's Make (as called in the top-level 'project' script) we completely close-off the host's environment and have our own PATH. So 'env PROGNAME' should expand to the PROGNAME in Maneage, right? Even if 'env' doesn't belong to Maneage itself.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Tue 09 Jun 2020 06:21:58 PM UTC, comment #2: 

Looking at this LWN article - https://lwn.net/Articles/779997/ - which is pointed to by the direct Wikipedia article - https://en.wikipedia.org/wiki/Shebang_(Unix) - "so well written, such that they use the given 'SHELL' environment variable in their shebangs" suggests not just a reproducibility bug, but a potential install bug on a GNU/Linux system.

If someone has lots of subdirectories and installs a 'maneage' system in, let's say, a 12-th level subdirectory, where the average character length of a directory name is 9 characters, then that's already a 120-character string. Add in #! at the beginning and software/installed/bin/bash at the end, and the line will be over 128 characters. Which will almost certainly lead to fatal errors if egrep, fgrep, pod2texi, or texi2any are used.

Such as situation will probably be rare in practice, but it's the sort of limit that's useful to keep in mind - just in case.

Having to write .local/bin/<executable> everywhere seems like a very heavy solution. Would doing this be worth it to increase portability and reproducibility? The use of a relative path would  make the script only usable from maneage root directory. For some specific cases this could make sense.

https://en.wikipedia.org/wiki/Shebang_(Unix) points out that "A common value is /bin/sh, but some systems such as Solaris have the POSIX-compatible shell at /usr/xpg4/bin/sh." It also suggests using #!/usr/bin/env sh .

Certainly #!/usr/bin/env python or #!/usr/bin/env perl is safer than using these directly - these are less fundamental languages to *NIX-like systems than bash .

Advice from people involved in comparing OSes and portability would probably be useful here.

Boud Roukema <boud>
Project Member
Tue 09 Jun 2020 02:16:49 AM UTC, comment #1: 

This is indeed an important point! Thanks for bringing it up ;-)!

I just tried it (changing '.build/software/installed' to '.local') and indeed found many such scripts!

$ for i in $(file .local/bin/*|grep script |awk '{print $1}' |tr -d ':'); do printf "${i} ::: " ; head -n1 $i ; done
.local/bin/astscript-sort-by-night ::: #!/bin/sh
.local/bin/autopoint ::: #! /bin/sh
.local/bin/bashbug ::: #!/bin/sh -
.local/bin/bzdiff ::: #!/bin/sh
.local/bin/bzgrep ::: #!/bin/sh
.local/bin/bzmore ::: #!/bin/sh
.local/bin/c_rehash ::: #!/usr/bin/env perl
.local/bin/curl-config ::: #! /bin/sh
.local/bin/dvipdf ::: #!/bin/sh
.local/bin/egrep ::: #!/my/build/directory/software/installed/bin/bash
.local/bin/eps2eps ::: #!/bin/sh
.local/bin/fgrep ::: #!/my/build/directory/software/installed/bin/bash
.local/bin/gettextize ::: #! /bin/sh
.local/bin/gettext.sh ::: #! /bin/sh
.local/bin/git-cvsserver ::: #!/usr/bin/perl
.local/bin/glibtool ::: #! /bin/bash
.local/bin/glibtoolize ::: #! /bin/sh
.local/bin/gsbj ::: #!/bin/sh
.local/bin/gsdj ::: #!/bin/sh
.local/bin/gsdj500 ::: #!/bin/sh
.local/bin/gsl-config ::: #! /bin/sh
.local/bin/gslj ::: #!/bin/sh
.local/bin/gslp ::: #!/bin/sh
.local/bin/gsnd ::: #!/bin/sh
.local/bin/gunzip ::: #!/bin/bash
.local/bin/gzexe ::: #!/bin/bash
.local/bin/libpng16-config ::: #! /bin/sh
.local/bin/lprsetup.sh ::: #!/bin/sh
.local/bin/ncursesw6-config ::: #!/bin/bash
.local/bin/pdf2dsc ::: #! /bin/sh
.local/bin/pdf2ps ::: #!/bin/sh
.local/bin/pdftexi2dvi ::: #!/bin/sh
.local/bin/pf2afm ::: #!/bin/sh
.local/bin/pfbtopfa ::: #!/bin/sh
.local/bin/pod2texi ::: #! /my/build/directory/software/installed/bin/perl
.local/bin/pphs ::: #!/bin/sh
.local/bin/printafm ::: #!/bin/sh
.local/bin/ps2ascii ::: #!/bin/sh
.local/bin/ps2epsi ::: #!/bin/sh
.local/bin/ps2pdf ::: #!/bin/sh
.local/bin/ps2pdf12 ::: #!/bin/sh
.local/bin/ps2pdf13 ::: #!/bin/sh
.local/bin/ps2pdf14 ::: #!/bin/sh
.local/bin/ps2pdfwr ::: #!/bin/sh
.local/bin/ps2ps ::: #!/bin/sh
.local/bin/ps2ps2 ::: #!/bin/sh
.local/bin/texi2any ::: #! /my/build/directory/software/installed/bin/perl
.local/bin/texi2dvi ::: #! /bin/sh
.local/bin/texi2pdf ::: #!/bin/sh
.local/bin/texindex ::: #!/bin/sh
.local/bin/uncompress ::: #!/bin/bash
.local/bin/unix-lpr.sh ::: #!/bin/sh
.local/bin/updatedb ::: #! /bin/sh
.local/bin/xml2-config ::: #! /bin/sh
.local/bin/xzdiff ::: #!/bin/bash
.local/bin/xzgrep ::: #!/bin/bash
.local/bin/xzless ::: #!/bin/bash
.local/bin/xzmore ::: #!/bin/bash
.local/bin/zcat ::: #!/bin/bash
.local/bin/zcmp ::: #!/bin/bash
.local/bin/zdiff ::: #!/bin/bash
.local/bin/zegrep ::: #!/bin/bash
.local/bin/zfgrep ::: #!/bin/bash
.local/bin/zforce ::: #!/bin/bash
.local/bin/zgrep ::: #!/bin/bash
.local/bin/zipgrep ::: #!/bin/sh
.local/bin/zless ::: #!/bin/bash
.local/bin/zmore ::: #!/bin/bash
.local/bin/znew ::: #!/bin/bash

Its interesting how some of them have already been so well written, such that they use the given 'SHELL' environment variable in their shebangs and have already accounted for this (some Texinfo and grep scripts, not all!).

I noticed that Gnuastro's 'astscript-sort-by-night' script also has this problem, I tried implementing a fix to it, but in the middle, I had second-thoughts: for example the "SHELL" environment variable my be set temporarily to some temporarily-built file. So if we use it in a permanently installed file, the program will crash. So I am not too sure now about its usefulness for individual programs. Its a higher-level decision for systems like Maneage.

We can add such a step in the 'reproduce/software/shell/configure.sh' script, after the building of software has finished.

We should just use our own installed programs in the loop (as we do in all the steps of 'configure.sh' after installing programs). For example '.local/bin/file' and etc, like this:

for i in $(.local/bin/file .local/bin/* | .local/bin/grep script | .local/bin/awk '{print $1}' | .local/bin/tr -d ':'); do .local/bin/printf "${i} ::: " ; .local/bin/head -n1 $i ; done

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Tue 09 Jun 2020 01:41:57 AM UTC, original submission:  

I suspect that both a lot of science-level software and some
system-level software uses scripts that have #!/bin/bash or
#!/bin/sh or #!/usr/bin/perl -w or #!/usr/bin/env python
in their shebang lines.

We could probably create a checklist lint type script to help
people find these and consider switching to the maneage
versions rather than the host versions. This would increase
the probability of both portability and long-term reproducibility.

The lint script should also add various other tests that we
find are useful. For the moment, this could be an example to
find some of these shebang scripts:

-verbatim-
for i in $(file .build/software/installed/bin/*|grep script |awk '{print $1}' |tr -d ':'); do printf "${i} ::: " ; head -n1 $i ; done
-verbatim-

Boud Roukema <boud>
Project Member

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    No changes have been made to this item

    Back to the top


    Powered by Savane 3.5