taskManeage - Tasks: task #15363, Better solution to restoring file...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15363: Better solution to restoring file dates after Git checkout

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Sat 10 Aug 2019 07:19:01 PM UTC
   
 
Should Start On:  Fri 09 Aug 2019 11:00:00 PM UTC Should be Finished on:  Fri 09 Aug 2019 11:00:00 PM UTC
Category:  Analysis Priority:  6
Status:  Postponed Privacy:  Public
Assigned to:  None Percent Complete:  0%
Open/Closed:  Open Effort:  0.00

Sat 17 Aug 2019 07:55:28 PM UTC, comment #1: 

I just noticed that this doesn't just apply to configuration files. Within a project, we will often set programming sources (in C, Python, or Bash) as prerequisites of steps also. This is commonly done so if anything changes in a script/source, the step that depends on them is re-run.

Therefore the modified solution should work on all the file in directories under `reproduce/analysis' except for `reproduce/analysis/make'.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 10 Aug 2019 07:19:01 PM UTC, original submission:  

One major problem with Git is that it doesn't keep the file meta data after a checkout (in particular the date). Git just copies a new file during checkout, so all files that have changed between branches get the current time stamp.

In most scenarios this is no problem, but for our template its a major problem because Make looks at timestamps, and we want our analysis outputs to depend on configuration files (in `reproduce/analysis/config/XXXXXX.mk'), so for example when I change the configuration settings for one step, I want Make to find out and re-do the analysis of that step is done.

The problem comes up when you checkout to the `template' branch to update your infra-structure, and then check back into your project's `master' branch, all your project-specific configuration files will be newer than your existing project outputs and Make will try to re-do the whole thing again!!!!

As a solution to this problem, we are currently using Metastore (actually my fork of it, because it is apparently not maintained for a long time). Metastore keeps this extra file info in a binary file (which we call `.file-metadata', and has `pre-commit' and `post-checkout' hooks to apply it for fixing the files.

But the fact that the file is binary is very bad, also since it keeps all file information, occasionally, it has problems, with directories and prints many warnings after a checkout, and sometimes it doesn't work on some files(!).

I think we can implement a more simple/elegant solution to this problem for this template.

Here is one suggestion: using Git `pre-commit' and `post-checkout' hooks (inspired from Metastore), but to only parse the contents of the `reproduce/analysis/config/*.mk' files and keep their dates (before committing them) as a specially formatted comment (line) in the same file. Then after a checkout we can look at all the checked-out files, read that special line and set its date based on that line.

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Digest:
   task dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-09-13 boud Dependencies- task #15942 is dependent

    Back to the top

    Powered by Savane 3.13-5884.
    Corresponding source code