taskReproducible paper template - Tasks: task #15363, Better solution to restoring file...


You are not allowed to post comments on this tracker with your current authentication level.

task #15363: Better solution to restoring file dates after Git checkout

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Sat 10 Aug 2019 07:19:01 PM UTC  
Should Start On:  Fri 09 Aug 2019 11:00:00 PM UTC Should be Finished on:  Fri 09 Aug 2019 11:00:00 PM UTC
Category:  Analysis Priority:  6
Status:  Postponed Privacy:  Public
Percent Complete:  0% Assigned to:  None
Open/Closed:  Open Effort:  0.00

Sat 17 Aug 2019 07:55:28 PM UTC, comment #1: 

I just noticed that this doesn't just apply to configuration files. Within a project, we will often set programming sources (in C, Python, or Bash) as prerequisites of steps also. This is commonly done so if anything changes in a script/source, the step that depends on them is re-run.

Therefore the modified solution should work on all the file in directories under `reproduce/analysis' except for `reproduce/analysis/make'.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sat 10 Aug 2019 07:19:01 PM UTC, original submission:  

One major problem with Git is that it doesn't keep the file meta data after a checkout (in particular the date). Git just copies a new file during checkout, so all files that have changed between branches get the current time stamp.

In most scenarios this is no problem, but for our template its a major problem because Make looks at timestamps, and we want our analysis outputs to depend on configuration files (in `reproduce/analysis/config/XXXXXX.mk'), so for example when I change the configuration settings for one step, I want Make to find out and re-do the analysis of that step is done.

The problem comes up when you checkout to the `template' branch to update your infra-structure, and then check back into your project's `master' branch, all your project-specific configuration files will be newer than your existing project outputs and Make will try to re-do the whole thing again!!!!

As a solution to this problem, we are currently using Metastore (actually my fork of it, because it is apparently not maintained for a long time). Metastore keeps this extra file info in a binary file (which we call `.file-metadata', and has `pre-commit' and `post-checkout' hooks to apply it for fixing the files.

But the fact that the file is binary is very bad, also since it keeps all file information, occasionally, it has problems, with directories and prints many warnings after a checkout, and sometimes it doesn't work on some files(!).

I think we can implement a more simple/elegant solution to this problem for this template.

Here is one suggestion: using Git `pre-commit' and `post-checkout' hooks (inspired from Metastore), but to only parse the contents of the `reproduce/analysis/config/*.mk' files and keep their dates (before committing them) as a specially formatted comment (line) in the same file. Then after a checkout we can look at all the checked-out files, read that special line and set its date based on that line.

Mohammad Akhlaghi <makhlaghi>
Project Administrator


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    No changes have been made to this item

    Back to the top

    Powered by Savane 3.5