taskReproducible paper template - Tasks: task #15345, Automatically using backup tarball...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15345: Automatically using backup tarball repository

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Mon 29 Jul 2019 01:08:10 AM UTC  
 
Should Start On:  Sun 28 Jul 2019 11:00:00 PM UTC Should be Finished on:  Sun 28 Jul 2019 11:00:00 PM UTC
Category:  Software Priority:  5 - Normal
Status:  Postponed Privacy:  Public
Percent Complete:  80% Assigned to:  None
Open/Closed:  Open Effort:  0.00

Tue 06 Aug 2019 01:15:10 AM UTC, comment #4: 

I started an early implementation of this on my fork's `auto-mirror' branch. The first commit shows my current implementation. In short, there is now a `mirror.conf' file that a project designer can define their own list of mirrors to use when a tarball can't be downloaded from its original URL.

I haven't had time to fully check it, so I haven't merged it into the project's `master' branch yet.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Mon 05 Aug 2019 12:47:39 AM UTC, comment #3: 

The current backup repository is in Gitlab, which can't be opened in some countries.

I have cloned that repository in this top-level address: `http://akhlaghi.org/reproduce-software' which doesn't actually have any HTML file (so it won't open anything if you click on it), but only contains the tarballs. For example this link for Gnuastro's tarball will work: http://akhlaghi.org/reproduce-software/gnuastro-0.10.tar.lz .

Because all the tarballs (including the ones that are only in our repository) are now available there, the base URL of those tarballs has now also been changed pointing to this URL.

Until we complete this task, if the download of any of the software has problems, just download them (with the same filename) from this top URL `http://akhlaghi.org/reproduce-software'.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Mon 29 Jul 2019 01:28:43 PM UTC, comment #2: 

Thanks for bringing this up Boud, you are completely right.

Checking tarball checksums is something I have been thinking of doing for a long time, but I haven't had the time to do it yet. But we will definitely add this step :-).

I just defined task #15347 to officially document this task. Hopefully one of us will be able to do it. As described there, its not too hard ;-).

In fact, even for Python packages, we don't check the checksums! Those hash values are actually within the URL that we use to download the tarball!

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Mon 29 Jul 2019 01:02:55 PM UTC, comment #1: 

Is it correct that as of commit da1123c, hashes to check tarball
validity are only provided for python packages?

grep "h=" reproduce/software/make/*.mk

only finds the python hashes. I'd agree that python packages are
probably the most likely to be risky, although lower level routines
are at risk of having exploits too. Access to a user shell is a big
step towards accessing root.

I think that a necessary element of automatically using backup tarballs
would be having checksums for these. Using a mix of different URLs forces
the user to implicitly associate a primary trust/confidence value to the individual or
group providing the overall package, since s/he won't be able to make
serious judgments on dozens of individual websites. This overlaps with the
general issue of "prefer-native-libraries" reproducibility versus exact reproducibility. The more that goes through a
trusted community with fully transparent well-defined pipelines and procedures
and decision making (such as Debian), the more trust the user can have in software
being secure.
https://en.wikipedia.org/wiki/Web_of_trust
https://en.wikipedia.org/wiki/Computational_trust
https://en.wikipedia.org/wiki/Trust_(social_science%29
https://en.wikipedia.org/wiki/Computational_trust
(A quote from the last of those four articles: _"Computational Trust applies
the human notion of trust to the digital world, that is seen as malicious rather than
cooperative."_ I think that's fair enough, in the sense that crackers' robots,
and incompetent annoying robots, are very much real in cyberspace.)

The python checksums appear to have 64 hexadecimal characters, with two pairs split off,
so my guess is that they're sha256sum checksums. Shifting to sha512sum would probably be
safer (disclaimer: I'm not an encryption expert).

Anyway, my recommendation would be to, at least, have checksums for all downloaded packages
that are not from the native distribution. A separate, though related, issue, would be
to have some sort of systematic link to security alerts/updates. Again, this is something
way beyond anything I'm experienced in.

Boud Roukema <boud>
Project Member
Mon 29 Jul 2019 01:08:10 AM UTC, original submission:  

When a software's server is in-accessible for any reason, and the template user doesn't have a local copy of their necessary tarballs, the pipeline will stop progressing.

We already do have a backup repository with all the necessary software's tarballs. But so far, its up to the user to manually get the tarballs from here and put it where the template can see it.

We should make this automatic: as soon as a software can't be downloaded from its original URL, the template goes to our backup repository and tries downloading it from there.

Mohammad Akhlaghi <makhlaghi>
Project Administrator

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-08-06 makhlaghi Percent Complete0% => 80%
    2019-08-05 makhlaghi Carbon-Copy- => Added surenaftm@gmail.com

    Back to the top


    Powered by Savane 3.5