mainstoreBackup - Support: sr #107261, Detect Bit-rot?


sr #107261: Detect Bit-rot?

Submitter:  None
Submitted:  Mon 08 Feb 2010 09:21:19 PM UTC
Category:  None Priority:  5 - Normal
Severity:  2 - Minor Status:  Done
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Sat 20 Feb 2010 12:34:00 PM UTC, comment #6: 

Thanks for your explanation, Heinz-Josef! For me, it's completely perfect to not calculate the md5sums for the source files with every run of storeBackup.
The option I'd be happy with is something like "checkForUnexpectedSourceChanges": Find all source files with mtime,ctime (size?) not changed since the latest (or any other) backup, and only for these source files, calculate the md5sum, compare it to the md5sum in the backup and report any mismatches. I'm not enough a computer expert to really confirm that this is "perfect data protection", but it's the best I can think of.

I see that there may be circumstances where changes are erroneously reported as "unexpected" (like the vfat thing), but I'd hope that these cases are rare and could be handled manually, i.e. "accepted as regular/expected change".

Of course, such a checkForUnexpectedSourceChanges run will take its time, but I don't have any better idea and also would only run this from time to time (like n-monthly?).

Thank you very much

Fri 19 Feb 2010 09:37:44 AM UTC, comment #5: 

storeBackup checks the file stamp (ctime, mtime) and the size to check if a file has changed. If these are exactly the same as before (stored in .md5CheckSums), no md5sum is calculated. This is because of performance. The very first version of storeBackup for my private use on a laptop (which was never published) has calculated all md5sums every time again and it did absolutely not scale for a lot of data.

You are right, up to now storeBackupCheckBackup does nothing to solve the problem, it just recognises them. One easy way for you to "solve" the problem would be to delete the line with that file from the .md5CheckSum file. This means for storebackup, that the file does not exist in the backup any more. Naturally, storeBackupCheckBackup still will report and error. To get rid of this, you have to delete the file itself also.

I have no idea why you got this bit errors. I never ever had them. From my understanding, the hard disk should use checksums to recognize / correct this errors. Or do I am wrong?
I think your hard disk in the linux box might be corrupt and has to be replaced. Whatever, it's not the right place to save a backup ;-) .

I think to check against bit faults in the source and in the backup, you have to run storeBackupCheckBackup and after this a version of storeBackup which calculates all md5sums (it should not be too difficult for me to add such an option). It also has to check if the time stamps and size are the same but the file is different, this might be a bit rot problem. (If you save files from vfat where Windows has written something, this sometimes happen without a bit rot :-( . And naturally, the time stamps have a granularity which might result in a change with the same time stamps in special situations.).

But that's all very slow naturally, backing up while calculating all md5sums generates the same load and takes the same amount of time like running storeBackupCheckBackup.

Heinz-Josef Claes <hjclaes>
Group administrator
Mon 15 Feb 2010 07:23:35 PM UTC, comment #4: 

Well, it looks like one of my backup machines has corruption problems again. I've got one backup machine running OpenSolaris so I can use ZFS, and one running Ubuntu 9.04, and the Ubuntu machine seems to have problems writing the data to disk without altering a bit every now and then.

If you are worried about bit-rot, consider storing your files on a machine running OpenSolaris so you can use ZFS and some replication. It checksums everything. It's not perfect, but it appears to be more robust that most common file systems.

Willard Korfhage <wkorfhage>
Mon 15 Feb 2010 05:34:33 PM UTC, comment #3: 

My backups are several terabytes, and the initial backup can take days backing up to a local file server. However, the daily backup on 4.5TB of generally unchanging files takes about 15 minutes, so it has to be checking the modification time before going the whole md5 route.

Doing a checkBackup takes hours. That reminds me I haven't run one in a while, so I'll do that right now.

> I understand the bzcat and diff part, but what are the "files of interest"? I'd say this are any files with unchanged timestamp, size etc. but changed md5sum, but how could grep help on this? Or would the grep part be based on my knowledge of which files must not change (like ~/fotos/*, for example)..?

Yes, grep was just to limit the list to directories you really care about. But if you want to know about all changed files, skip the grep.

Willard Korfhage <wkorfhage>
Mon 15 Feb 2010 04:35:12 PM UTC, comment #2: 

Thank you for your reply! If I got it right, storeBackup calculates the checksum for every source file every time it is run, no matter whether a files timestamp changed or not? Or how would it recognize an "unchanged" (timestamp, size...) file as "changed" without md5sum? Besides the effects of backing up bit-rotted files, I wonder how long storeBackup will work (i.e. run md5sum) on my 120GB home dir... (But run-time is not the #1 issue for me when talking about backup systems).

On watching for unexpected changes, you wrote:

> Bzcat, followed by grep to get the checksum lines for the files of interest, then diff against the previous day to check for changes.

I understand the bzcat and diff part, but what are the "files of interest"? I'd say this are any files with unchanged timestamp, size etc. but changed md5sum, but how could grep help on this? Or would the grep part be based on my knowledge of which files must not change (like ~/fotos/*, for example)..?

Thanks -

Mon 15 Feb 2010 03:59:15 PM UTC, comment #1: 

It would recognized the file as changed, and back up the new version. I've been (very slowly) working on a program that will analyze the checksum file for de-duping purposes, but with some additional work it could also be used to watch for unexpected changes. In fact, now that I think about it, just watching for unexpected changes ought to be a lot simpler. Bzcat, followed by grep to get the checksum lines for the files of interest, then diff against the previous day to check for changes.

StoreBackupCheckBackup certainly is worth running. It found that in one of my files, one bit was changed when the backup was written out to disk. Unfortunately, it is not much help beyond that detection. Because the source file never changed, in subsequent backups, storeBackup merely linked to the corrupted file. Even moving the file around would not help, because storeBackup would identify it by the md5 and, again, link to the corrupted file. If I had caught it quickly, I could have just deleted the backups for a day. As it was, I ended up discarding and rebuilding that whole backup repository because I had another backup elsewhere.

Willard Korfhage <wkorfhage>
Mon 08 Feb 2010 09:21:19 PM UTC, original submission:  

First of all: I'm really impressed by storeBackup's wealth of features, options and documentation! I am going to install and use it, but have one additional question in mind. It may be paranoid, but that's what every backup is more or less:-)

With a good backup setup, I'm no longer afraid to loose data by accidential deletion or hardware failure. But I'm still afraid of bit-rot, especially in my collection of digital images. Those files are never changed and only read in very rare cases. Thus, if such a file is damaged/corrupted by bit-rot or any other cause, I would not recognize it, possibly for years. Its size or timestamp would probably not change.

That is where storeBackup might come in. I understood that storeBackupCheckBackup checks the md5sums versus the backup files in order to verify that the backup files are not corrupted. This would detect bit-rot on the backup disc - good!

But what about a file being corrupted on the source disk? Would it be recognized by storeBackup as "changed" because its checksum changed, even with the timestamp and file size being unchanged? That would be the worst case - the corrupted file would be backed up and if, in some years, I recognize the error, only useless backups of the corrupted file would be left.

That is why I wonder whether there's a way to check the md5sums against the source files? For every file that has not "officially" changed (timestamp?), I'd like to recalculate the md5sum and check it against the stored checksum. In case of mismatch, I would be warned "the content of this file changed where it shouldn't have" and could fix the issue by restoring the file from backup.

I hope you get my point... Maybe the "check md5sum against source file" thing isn't a job for a backup software. But storeBackup has everything that is needed... Anyway, what's the criteria for storeBackup to regard a file as "changed"? Would storeBackup detect a corrupted, but "unchanged" file as changed via checksum?

Thank you very much in advance for your thoughts and hints on this issue...




(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)
  • -email is unavailable- added by wkorfhage (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-05-31 hjclaes StatusNone Done

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code