bugstoreBackup - Bugs: bug #39994, An error happen when I backup data.

 
 

bug #39994: An error happen when I backup data.

Submitter:  None
Submitted:  Tue 10 Sep 2013 03:12:55 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 30 Sep 2013 12:40:02 PM UTC, comment #3: 

fixed in 3.4.2

Heinz-Josef Claes <hjclaes>
Group administrator
Mon 16 Sep 2013 02:01:08 PM UTC, comment #2: 

Hi, all I wrote below is correct.

BUT - seems I just found a bug when trying to handle the issues.

Because this kind of bug happens heuristically I have to check for some time in my backups. If you want, send me an email so I can mail you the fix.

Regards, Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Tue 10 Sep 2013 08:17:33 PM UTC, comment #1: 

Hi, that's complicated. I'll try to explain what's happening.

storeBackup.pl works (among others) in the following way:

1. check by timestamp, size, path/filename if a file is already in the backup. If not:

2. calc all (or lots of) md5 sums from new / changes files in a directory by calling the program md5sum. This is done in parallel to other stuff (mostly copying, compressing, hard linking, reading directories)

3. put the md5 sums in a waiting queue for compression or a queue for copying (let's say for compression)

Now, there is a problem:
If the file (which name was picked up from the queue) changed since the time of calculation of the md5 sum, then the md5 sum in the file in the backup will be different from the one calculated with md5sum before (the change) and stored in .md5CheckSums.

Therefore:
4. an external program (lib/stbuMd5Exec.pl) is started in parallel (to other activities) which recalculates the md5sum of the data and compresses the file simultaneously reading the file only once. This program writes it's results (md5sum + size of the file) to a temporary file which is read by storeBackup.pl. And the reading of that temporary file is not successful.

The reason for this is the following issue:
If you start an external program (fork/exec) and wait for it to finish, then you cannot always be sure to be able to read a file the external program wrote. This may be due to buffered writing or whatever. In these very rare cases storeBackup.pl waits some time and if that's not enough finally calls sync. But in some even more rare cases this also doesn't help. Then the message you saw is written to the log file.
The same sometimes happens with redirected output (stderr, stdout).

I think this may be an issue of buffering and/or caching!?

I was never able to really reproduces these rare cases. Inserting a print statement is enough to change the behaviour.

The only answer I can give you: simply ignore the message.

Maybe this helps?


PS: If somebody knows how to handle this problem .... would be nice to hear.

Heinz-Josef Claes <hjclaes>
Group administrator
Tue 10 Sep 2013 03:12:55 AM UTC, original submission:  

when I use storebackup tools to backup data,I found an error in the logs.

ERROR     2013.09.10 05:12:59 23793 cannot read recalced md5sum of </home/mysql/default/2013.09.10_04.07.13/test/file_453.ibd>; file is not backed up (3)

But I found '/home/mysql/default/2013.09.10_04.07.13/test/file_453.ibd.gz' file and its md5 is match the source file.

Please help me to find out why there is an ERROR.
Thank you!

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-09-30 hjclaes StatusNone Fixed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code