mainstoreBackup - Support: sr #107130, Proper strategies for dealing with...


sr #107130: Proper strategies for dealing with md5 sum mismatches?

Submitter:  Willard Korfhage <wkorfhage>
Submitted:  Tue 17 Nov 2009 10:30:01 PM UTC
Category:  None Priority:  5 - Normal
Severity:  4 - Important Status:  Done
Privacy:  Public Assigned to:  None
Open/Closed:  Open Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup

Wed 18 Nov 2009 03:13:29 PM UTC, comment #3: 

I forgot to add, thanks very much for your help. I really want to make storeBackup work because some of my files are over 1 TB (video files), and I can't afford to go having duplicates of those files in the backup archive.

Willard Korfhage <wkorfhage>
Wed 18 Nov 2009 03:04:42 PM UTC, comment #2: 

I have only been using storeBackup for 3 weeks, so I have always run the latest version. The files are many different types, but they seem to be larger files - mp3, wav, tiff, jpg, iso, etc. Only scattered files of these types have problems, not all of them (250 or so out of hundreds of thousands). No text files appear with incorrect checksums.

One thought: these appear to be the same types that storeBackup is not supposed to compress. My config file says

addExceptSuffix= '\.JPG' '\.GIF' '\.PNG' '\.jp2' '\.JP2' '\.dng'
        '\.flac' '\.mpc' '\.wma' '\.WMA'
        '\.avi' '\.AVI' '\.DIVX' '\.divx' '\.XDIV' '\.xdiv' '\.wmv' '\.WMV'
        '\.mov' '\.MOV' '\.dvr-ms' '\.tp' '\.mp4' '\.MP4'
        '\.tib' '\.gho' '\.GHO' '\.GHS' '\.iso' '\.ISO'

I can understand that there might be a problem if the file changes while being backed up, but these are file that haven't changed, sometimes for years. My big concern is that when I delete a backup, backups that previously checked out ok now have errors, so I have no idea if any of my backups are valid or not.

Perhaps unrelated, I also get some of the following error messages when I run the backup check

Use of uninitialized value $rel in hash element at /usr/local/bin/ line 846.
Use of uninitialized value in numeric eq (==) at /usr/local/bin/ line 846.

I get 16 of these message pairs, which is much smaller than the number of files with mismatching md5 checksums. These came at the end of the run, just preceding the last info lines.

Willard Korfhage <wkorfhage>
Wed 18 Nov 2009 12:54:10 PM UTC, comment #1: 

Since the last version of storeBackup it handles md5sum more carefully.

The former versions calculated the md5sum of a file which was copied later. If the file changed during that time (normally a few seconds, perhaps minutes), the md5sum in the repository (file .md5CheckSum) did not match the file in the backup. This could (mostly theoretically) lead to a "wrong" next backup.

Now, storeBackup calculates the md5sum (like before) to decide to copy or link it. When copying / compressing the file, at the same time it calculates the md5sum (one read process) to make sure the file has not changed. So - if it changed, storeBackup will report you a changed md5sum (it reports "file <xyz> changed during backup").
When hard linking the file, this simply reflects the situation when the md5 sum calculation was done (even if it changed a microsecond later).

When running storeBackupCheckBackup, you will probably find mismatches because files have changed (in the old versions of the storeBackup).

Are you reporting this mismatch while running the latest version of storeBackup always? If yes, what kind of files does storeBackupCheckBackup report?

After running storeBackupUpdateBackup (successful), the result is absolutely the same as running storeBackup without option --lateLinks. So no problem to use 'rm -rf'.

Heinz-Josef Claes <hjclaes>
Group administrator
Tue 17 Nov 2009 10:30:01 PM UTC, original submission:  

I am not sure how to deal md5 sum mismatches that storeBackupCheckBackup finds. I have tried just deleting the backups for the days with the mismatch, but that just seems to propagate the problem to the next backup, and makes me distrust the entire system. Specifically, I had a series of daily backups, and the first two registered errors in the backup check, but nothing in the others. I deleted those two backup directories, ran the backup again, and then third day suddenly showed up as having md5 sum mismatches. I deleted that, and then the 4th day, which had been fine, now had problems, and so on.

Did just doing rm -rf on that day's subdirectory break other parts of the backup? The docs say it is ok if not using latelinks. I have used latelinks, but also made sure that storeBackupUpdateBackup ran before I did the removal.

Willard Korfhage <wkorfhage>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)
  • -email is unavailable- added by wkorfhage (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-05-31 hjclaes StatusNone Done

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code