taskManeage - Tasks: task #15708, Only connecting to Zenodo when...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15708: Only connecting to Zenodo when necesssary

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Sun 28 Jun 2020 12:09:54 AM UTC
   
 
Should Start On:  Sat 27 Jun 2020 11:00:00 PM UTC Should be Finished on:  Sat 27 Jun 2020 11:00:00 PM UTC
Category:  Software Priority:  5 - Normal
Status:  Postponed Privacy:  Public
Assigned to:  None Percent Complete:  0%
Open/Closed:  Open Effort:  0.00

Tue 30 Jun 2020 08:17:05 PM UTC, comment #2: 

Good point! Yes! The DOI is the more trustable link (in case Zenodo changes its internal file structure later!).

But its not as straightforward as seeing if a software directory is given because the software directory may not be complete (and the project's software versions may have been updated since the last time that directory was updated.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 30 Jun 2020 06:20:55 PM UTC, comment #1: 

There's not only a connection to Zenodo, there's first a connection to doi.org :

Resolving doi.org (doi.org)... 172.67.72.147, 104.26.8.237, 104.26.9.237, ...
Connecting to doi.org (doi.org)|172.67.72.147|:443... connected.
HTTP request sent, awaiting response... 302
Location: https://zenodo.org/record/3883409 [following]


This also has to be tidied up and checked for users who start out with their own tarballs. One of the officially claimed [1] features of maneage is that a maneaged project can be reproduced completely offline. This should be straightforward to handle: if the user has set a local tarball directory, then checking at Zenodo:3883409 is disabled, together with all other downloading.


[1] https://arxiv.org/abs/2006.03018

Boud Roukema <boud>
Group Member
Sun 28 Jun 2020 12:09:54 AM UTC, original submission:  

Currently, every time we run './project configure', it will attempt to connect to Zenodo and find the most recent upload to use in possible downloads.

But especially in the development phase, this is really annoying and slows down the testing. It can also cause confusion for a user who has all their project's software already in a directory and doesn't want to connect to the internet.

Fortunately the software downloading phases have become highly modular and all downloads (except for those in 'pre-make-build.sh') are organized in a singular place now. They are also 'flock'ed (to avoid parallel downloads). So we can move this check within that part of the script and only connect to Zenodo for a simple check when its necessary.

The resulting Zenodo URL can be written to a file which is later checked by other download attempts. Also, in the 'configure.sh' script (before the first download attempt) we can delete this file (if it exists) to make sure that the same Zenodo URL is not used in two rounds of configuration (possibly separated by months!).

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code