bugmonotone - Bugs: bug #11145, save and restore ctime and mtime

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #11145: save and restore ctime and mtime

Submitter:  None
Submitted:  Sat 27 Nov 2004 06:33:25 AM UTC
Votes: 70
 
Category:  None Severity:  1 - Wish
Item Group:  feature request Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Closed
mtn version --full: 


Jump to the original submission

Wed 19 Jul 2006 05:35:53 PM UTC, comment #10: 

quote: Files that were modified by the update, keep their old mtime.

I assume you ment to say NOT modified ?
This behavior is different/better than subversion, which always modifies the mtime, as far as I can tell, from the testing that I did.

-------------

quote:  (I hope win32 does not call file creation time "ctime", that would make things even more horribly confusing.)

Ah, no wonder I am so confused.  In the windows api ctime does refer to "creation time". I did assume that in Linux, references to ctime meant CREATION time.  Below are excerpts from  the windows api manual. 

Thank you for being so patient and explaining this to me, I appreciate it.  I am ignorant not stupid, there is a difference.

I am puzzeled that Linux lacks this property/concept.  I'm fairly new to *nix, and don't claim to be a file systems expert, but every other file system I've delt with from the Heath H8 on up, has had this concept. People like to know when something was created and windows goes to great lengths to preserve this value.


-------------------------
Excerpts from Windows api manual:

The _fstat function obtains information about the open file associated with handle and stores it in the structure pointed to by buffer. The _stat structure, defined in SYS\STAT.H, contains the following fields:

st_atime  -  Time of last file access.

st_ctime  -  Time of creation of file.

st_mtime  -  Time of last modification of file.

--------------------

The STATSTG structure contains statistical information about an open storage, stream, or byte array object. This structure is used in the IEnumSTATSTG, ILockBytes, IStorage, and IStream interfaces.


typedef struct tagSTATSTG
{
    LPWSTR          pwcsName;
    DWORD           type;
    ULARGE_INTEGER  cbSize;
    FILETIME        mtime;
    FILETIME        ctime;
    FILETIME        atime;
    DWORD           grfMode;
    DWORD           grfLocksSupported;
    CLSID           clsid;
    DWORD           grfStateBits;
    DWORD           reserved;
} STATSTG;
 


mtime  -  Indicates the last modification time for this storage, stream, or byte array.

ctime  -  Indicates the creation time for this storage, stream, or byte array.

atime  -  Indicates the last access time for this storage, stream or byte array.

--------------------

{In the windows api, all file time properties are settable; but only on files, not folders}

The SetFileTime function sets the date and time that a file was created, last accessed, or last modified.

BOOL SetFileTime(
  HANDLE hFile,                      // handle to the file
  CONST FILETIME *lpCreationTime,    // time the file was created
  CONST FILETIME *lpLastAccessTime,  // time the file was last accessed
  CONST FILETIME *lpLastWriteTime    // time the file was last written
);


The FILETIME structure is a 64-bit value representing the number of 100-nanosecond intervals since January 1, 1601.


Erik MT <codehappy>
Group Member
Wed 19 Jul 2006 05:39:54 AM UTC, comment #9: 

Re: wikipedia: the table looks accurate to me.  A common point of confusion about unix file times is whether "ctime" has anything to do with file creation; it does not.  It tracks the last time the file's metadata was changed (e.g., changing its permissions or setting the mtime manually are two of the things that cause ctime to be updated).  That might explain some of our confusion.  (I hope win32 does not call file creation time "ctime", that would make things even more horribly confusing.)

> ...if I do a checkout or update and all the files get set to
> the timestamp of the update/checkout...


^^ If you do an update, then all the files that were updated get their mtime set to the time of the update.  Files that were modified by the update, keep their old mtime.

It may be helpful in thinking about this to distinguish between two "sorts" of files -- the actual objects on disk that the OS knows about ("concrete files"), and the abstract objects with history stored in the SCM repo ("abstract files").  A workspace consists of a bunch of concrete files that in some sense "represent" some particular abstract files.  So the question here is not "should workspaces have mtimes set?", the question is "should the mtime on a concrete file in the workspace match the last-modified time of that concrete file, or the last-modified time of the abstract file it represents?".  Since 'make' reads concrete files, the important thing to it is to know when concrete files have changed.

Similar comments apply to the dreamweaver case -- if the person who doesn't use any SCM always syncs with the website, and the website is a single workspace, well, then things will work fine even if your SCM does not track mtimes.  In fact, since dreamweaver only knows about concrete files anyway, they might work better.

Hope that clarifies what I was saying.

Nathaniel Smith <njs>
Group Member
Tue 18 Jul 2006 10:24:14 AM UTC, comment #8: 

in response to Nathaniel:

on the subject of make -- quote:  (in which case the time stamps on disk track the actual changes made to the actual files)

Now I am really confused.  if I do a checkout or update and all of the files get set to the timestamp of the update/checkout.  then the mtime is not tracking changes made to the files.  if all the dates have changed, this tells make that everything needs to be rebuilt.  or else nothing needs to be rebuilt, depending on the interpetation of what is current.  But if the timestamp shows the true last mod time -- barring timezone shifts etc -- then make knows which files were actually modified and only needs to rebuild those.

In the case of dream weaver sync.  the "working copy", is the actual website.  the goal of scm in this case is for history tracking and undo capability, plus possible mirroring to other locations.  The ability to track and undo changes to a website is  of tremoundous value.  unlike software development, there is typically only 1 person who works on any given web page at a time.  with or without scm, clock skew is a problem and therefore any environment which uses file times for sync must ensure the computers are in sync.  thus scm does not need to be unduely burdened with such concerns.

quote: No scm can support ctime tracking; there is no API (on posix anyway, dunno about windows)

ah well, I guess it is just part of my ignorance of Linux and the transistion to it.  Yes, on windows the ctime is settable except for folders.  Backup programs use this ability.  But ctime is mostly for trivia of "when did I create that file?", nice to have but not essential.  But the real problem I'm trying to solve is external sync;  there are many programs, especially in the windows world, that do the syncing based on mtime.  yes it can be fragile, but it generally is a good enough solution.  it really goes back to the admin taking responsibility for ensuring correct time sync of the cpus -- which is required for AD logins anyway.

quote: Tracking timestamps will create clutter in history (e.g., you touch a file to force make to rebuild it, and now your next commit includes a pointless timestamp update to that file)

If the file has not been modified, as determined by it's hash, then the change to the mtime is bogus and could be ignored.  only when the hash changes is the change to the mtime relevant.

Thank you again for the response.  That alone tells me a lot about the value of monotone,


Erik MT <codehappy>
Group Member
Tue 18 Jul 2006 09:38:14 AM UTC, comment #7: 

P.S.  I looked at the wikipedia article.
http://en.wikipedia.org/wiki/Filesystems#Metadata

and it was very interesting to see how many different file systems there are/were. 

However close examination of the table which lacks a single colume that is "all green"....  shows that the table is either wrong or misleading or both.  to begin with the concept of mtime is either very confusingly portrayed, or not portrayed at all ("Last Metadata Change", what the heck?).  and ctime appears to just be plain wrong.  for instance, it appears to claim that extN file systems do not support ctime.  I was really puzzeled by that.

Wikipedia is a great and wonderful thing, but the sad fact is, it must be taken with a large grain of skepticisim, it is not authoritative as even they will tell you.


Erik MT <codehappy>
Group Member
Tue 18 Jul 2006 06:58:23 AM UTC, comment #6: 

A few quick notes:
  -- In the Dreamweaver case, do you really need to be able to make a checkout from the scm, sync with that checkout, then make another separate checkout, and then sync with that checkout?  It seems like the workflow the Dreamweaver sync tool is designed for (must be designed for, or it wouldn't work!) is one where you only ever have one place you use it to sync with.  Whether or not your VCS stores dates does not affect such a workflow at all.  (In fact, it's likely that storing dates in the SCM could actually degrade such a system -- say you make some changes locally, push those to the server with the SCM, there's clock-skew between you and the server, and so the file on the server gets its date set backwards by your change.  Someone uses Dreamweave to sync with the server, Dreamweaver sees that the server has an "old" version of that file, and clobbers it.)

  (-- This reminds me, you mention 'make' as a tool that benefits from VCS tracking of file times.  This is incorrect; 'make' works as designed when the VCS does not track file times (in which case the time stamps on disk track the actual changes made to the actual files), and can fail mysteriously if the VCS does (in which case the time stamps on disk track the clock vagaries of whatever other computers work is done on.)

  -- Tracking timestamps will create clutter in history (e.g., you touch a file to force make to rebuild it, and now your next commit includes a pointless timestamp update to that file), potentially a great deal of clutter.  Such clutter, aside from making history harder to understand, can create spurious merge conflicts that must be resolved by hand.

  -- Using time stamps to do mirroring has always been a fragile hack; I suspect part of the reason people in the FOSS community are uninterested in supporting it is that, well, we all use rsync these days, because it's more reliable, faster, and immune to clock issues :-).

  -- No scm can support ctime tracking; there is no API (on posix anyway, dunno about windows) for setting the ctime of a file.  So I hope this isn't a requirement you really have.

This isn't meant to discourage you from finding a way to get this functionality with a VCS; you're the one who knows what you need :-).  And as Graydon says, we'd be perfectly happy to see a monotone add-on that did this, and have tried to make the appropriate hooks available.  But maybe it will help make more comprehensible to you, how it could be that some people think tracking timestamps is a bad idea -- even for an individual, ignoring the "big picture" :-).

Nathaniel Smith <njs>
Group Member
Tue 18 Jul 2006 05:18:25 AM UTC, comment #5: 

Thank you for the explaination.  Apologies if I came across sounding like a hot-head.  am just pretty frustrated from having spent so much time and energy trying to come up with an scm soltuion that is going to work for me.   All I want to do is get back to work writing code.  But my project has gotten too big to manage without scm.  and I am spending days and days, reading reviews and manuals on what should have been a trivial fundamental.

---------------

I see that the divergance is due to differences in perspective.  You are taking the lofty view (as you must) of many operating systems and uncontroled environments.  Whereas I am focused on what I need that will work for me in my circumstance, without giving sufficient consideration to the bigger picture.

As far as date validity goes....  if the truth be knowen, I don't care what the ctime/mtime is in absolute terms.  But I do care that it be consistantly maintained on my local system.  Suppose that I changed the file "yesterday".  When I change it today, it is mildly interesting for me to know in absolute terms when "today" is; but in relative terms I certainly want to know that "today" is more recent than "yesterday".  

Naturally the onus is upon me to maintain an accurate clock.

Consider this (real) example:
I admin a web server. I want to put the site under scm.  There would be many advantages to this.  But I don't maintain the site.  The person who maintains it, uses dream weaver which syncs the files based on file dates.  I doubt that I would be successful in getting this person to install/use an scm client.  Therefore the file dates must be preserved (relative to that user).  It doesn't matter what the dates are in absolute terms.  but it is vital that the dates not be changed.  

There are many other examples where unexpected changes to the date cause problems, such as site mirroring.

Bottom Line is that the mod date is set controlled outside of/independent of the scm.  it is not the scm's job to care what these date values are, but it should be the scm's job to preserve the dates to the greatest extent possible.  Naturally there will be situations where this can't be done, but it's up to the user to be cognizent of these limits and avoid the scenarios.

Your point about metadata inconsistancy is well taken.  But I can't think of a single os that does not support ctime and mtime for files.  I am very aware that Windows does not support setting the dates for folders, but folder dates are not pertinent to any sync systems that I am aware of. 

I guess the issue primarily revolves around compatibility of scm with various external sync methodologies.  Without this compatibility I simply can't use it.

At this point though, I'm thinking that I can cobble something together, along the lines of a previous suggestion -- for a seperate file to track the dates. But it means still another substantial expenditure of time on a task that is keeping me from my real work. 

Over-all I really like what I've seen so far of monotone.


Thank you for the response!!!

Erik MT <codehappy>
Group Member
Mon 17 Jul 2006 06:42:22 AM UTC, comment #4: 

Wow, I'm surprised about the vigorous opinions on the matter! I hope I can elaborate on the topic a bit without stepping on any toes; I stand by my initial statement that I'd at least be happy to integrate patches implementing this sort of functionality via attributes (including patches to extend how attributes work). I just want to explain why things are the way they are now.

File dates are really unreliable. If you're on a networked filesystem, they are almost guaranteed to be inconsistent. If your clock moves due to network time reset, drifting local clock, or manual clock adjustment your current notion of time may differ from your notion of time in the recent (real) past; you may give the impression that there have been real date changes when there have been none (or vice-versa), or even give the impression that you're moving backwards in time.

Furthermore, the set of dates recorded by each filesystem is not even consistent: note for example that in http://en.wikipedia.org/wiki/Filesystems#Metadata there is no column that's "all green".

So when we were doing early work on monotone we just skipped the whole topic of storing metadata. We included a general mechanism for users to add any metadata they felt important, and left it at that. There were more interesting experiments to spend our time on; we figured if users cared about some particular metadata they could spend some of their own time.

(Consider other metadata if you're offended by this perspective: there are just as many people who think it's "essential" to preserve file owners, file ACLs, extended file types, etc. None of which are reliable or portable either.)

I think most SCM tools -- especially the portable ones -- draw a line somewhere and say "we're not even going to bother trying to model that, it's more trouble than it's worth". CVS is much worse: they don't even model directories or renames, or logical changesets to multiple files. It's a question of prioritization and expenditure of energy, both in the initial writing of code and the inevitable increase in maintainance you'll have to do if you support it.

graydon hoare <graydon>
Group Member
Mon 17 Jul 2006 03:30:54 AM UTC, comment #3: 

Also relevant: http://lists.gnu.org/archive/html/monotone-devel/2006-07/msg00207.html , and particularly Graydon's reply.

Timothy Brownawell <tbrownaw>
Group Member
Mon 17 Jul 2006 02:56:04 AM UTC, comment #2: 

er uh, I'm confused by the LACK of this functionality.  What is it that I am not understanding?  I gave up on subversion because it lacked this - but especially because the subversion dev's had an attitude about even considering that this be added.

At least here I am glad to see there is receptivity to the concept.  So I guess I will hang around and give monotone a chance.  But this is potentially a deal killer for me.

But what the heck is up with this?  I am new to version control on Linux, am coming from Windows and Visual Source (un)Safe, where date preservation is a given.

There is a heck of a lot of stuff that cares about those dates.  DreamWeaver sync, for one, make and wget for another.  And me for a 3rd, When I move my currently uncontroled src files into scm I want to know what version they were, (which I track by their mtime)  

How could the major SCMs for Linux (cvs, subversion, monotone) so totally not get this concept/need.  Doesn't anyone sync their websites and put them in scm.  Doesn't anyone care what their code ctime/mtime was before putting it under scm?

Oh great sages, would you please enlighten this humble little grasshopper....    Thanks.


Erik MT <codehappy>
Group Member
Wed 26 Apr 2006 12:28:22 AM UTC, comment #1: 

This would probably use attributes now. Difficulties: mtime would have to be automatically updated (by a hook or something), while AIUI attrs can only be modified manually ATM. We'd need a special merger for these attrs, too.

Timothy Brownawell <tbrownaw>
Group Member
Sat 27 Nov 2004 06:33:25 AM UTC, original submission:  

when a revision is committed presently the ctime and mtime are not saved later when the revision is checked out these cannot be restored the files in that revision on disk appear as brand-new with a current date and not their appropriate dates which should be earlier than the time that revision was checked in

to solve this I propose to be compatible with previous versions I propose a new versioned metadata file named ".mt-dates" in which rows of three columns pathname mtime ctime are stored

when a commit occurs the file's dates are collected in this file and then the committee proceeds as normal

when a checkout occurs if the ".mt-dates" file exists after the files have been and disk and the checkout is almost complete that file is scanned and those dates are returned to their appropriate owners

this differs from bug #8465 in that it solves the right problem and does not have the nasty side effects of updating the ctime's of all the files inappropriately the key difference is when and what date is restored in this case we save and restore the creation time and modification time of the file that is given to it by the operating system

in bug #8465 it says "It would be nice if monotone would set the creation-time of files under it's control to the date and time of their last commit when checking out a branch."

this is wrong it is robbing a file of its appropriate modification date you should be a tell when a file was modified just by looking at the modification date

"Also, when an update of a file results in an unmodified copy of the file, it should set the cdate to the date of the last commit to this file."

this is just plain weird it's as if more information is trying to be encoded in the simple dates than should be

"The reason I'm asking for this is because I'd like to use this information in my homepage to display the file's ctime as "last modification date" on the web page, but that doesn't work at the moment, because I edit the files on a different machine than the web server and a "monotone update" will use the current time, not the time of the last commit."

you are using the wrong date its modification date after creation date which you should be using

so as we see my solution of saving and restoring the appropriate pair of dates to the files at the appropriate times is what is needed and not this insanity of trying to encode more than is necessary

so simply I state it again I wish that monotone would save and restore the appropriate creation time and modification time from and to the files

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tbrownaw (Updated the item)
  •  

    There are 70 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-11-25 tbrownaw Open/ClosedOpen Closed
    2006-07-17 tbrownaw Severity3 - Normal 1 - Wish
    2006-07-17 codehappy Carbon-Copy- Added codehappy

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code