mainstoreBackup - Support: sr #108963, What could be the cause for little...

 
 

sr #108963: What could be the cause for little file changes with many MD5 recalculations?

Submitter:  None
Submitted:  Thu 04 Feb 2016 01:18:46 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 08 Feb 2016 02:11:25 PM UTC, comment #2: 

Hello Heinz-Josef,

thanks for the info. I've checked some files and I think I have found the cause for the problem.

One part of the backup is a rsync job which does a rsync backup of many small files (about 500.000 attachment files of Zarafa Groupware) to a network share. The rsync job creates folders for several days in which the files are hard linked, to save space.

Now storeBackup backups this folder to an external disk. It seems that for all these hard linked files the md5 sum is calculated every time for new.

Now, I have done the backup without those attachment files and it finished a lot faster (only about 30m).

Thanks again for your help!

Anonymous
Fri 05 Feb 2016 11:25:08 AM UTC, comment #1: 

Hi,

the cause for this should be that one (or multiple) values for ctime, mtime or size of the file are different to the values stored in the previous backup.
You can look into the file .md5CheckSums.bz2 inside the top level of each backup. You can use llt (part of storeBackup) to calculate the "real date" from epoch, eg:

$ /opt/storeBackup/bin/llt -e 1454461228
1454461228 -> 2016.02.03 02:00:28

Does ctime / mtime change?


Heinz-Josef Claes <hjclaes>
Group administrator
Thu 04 Feb 2016 01:18:46 PM UTC, original submission:  

Hello,

in one storeBackup job that I have configured I can watch that although there were very little files changed, more than half of the MD5 sums have been recalculated which causes the job to take a very long time. Is there an easy fast way to find out, what could be the cause for that?

Thanks in advance for any hint.

Here is a sample statistics of the backup job:

INFO      2016.02.04 07:07:22  2427 syncing ...
STATISTIC 2016.02.04 07:07:25  2427  [sec] |      user|    system
STATISTIC 2016.02.04 07:07:25  2427 -------+----------+----------
STATISTIC 2016.02.04 07:07:25  2427 process|   1094.39|   3334.84
STATISTIC 2016.02.04 07:07:25  2427 childs |   1114.04|   3259.16
STATISTIC 2016.02.04 07:07:25  2427 -------+----------+----------
STATISTIC 2016.02.04 07:07:25  2427 sum    |   2208.43|   6594.00 => 8802.43 (2h26m42s)
STATISTIC 2016.02.04 07:07:25  2427                    directories = 215766
STATISTIC 2016.02.04 07:07:25  2427                          files = 2698918
STATISTIC 2016.02.04 07:07:25  2427                 symbolic links = 284442
STATISTIC 2016.02.04 07:07:25  2427                     late links = 0
STATISTIC 2016.02.04 07:07:25  2427                    named pipes = 267
STATISTIC 2016.02.04 07:07:25  2427                        sockets = 1497
STATISTIC 2016.02.04 07:07:25  2427                  block devices = 0
STATISTIC 2016.02.04 07:07:25  2427              character devices = 62
STATISTIC 2016.02.04 07:07:25  2427      new internal linked files = 62543
STATISTIC 2016.02.04 07:07:25  2427               old linked files = 2348084
STATISTIC 2016.02.04 07:07:25  2427                unchanged files = 2410627
STATISTIC 2016.02.04 07:07:25  2427                   copied files = 850
STATISTIC 2016.02.04 07:07:25  2427               compressed files = 1170
STATISTIC 2016.02.04 07:07:25  2427                  blocked files = 0
STATISTIC 2016.02.04 07:07:25  2427    excluded files because rule = 0 (0.0 )
STATISTIC 2016.02.04 07:07:25  2427    included files because rule = 0 (0.0 )
STATISTIC 2016.02.04 07:07:25  2427         max size of copy queue = 1
STATISTIC 2016.02.04 07:07:25  2427  max size of compression queue = 2
STATISTIC 2016.02.04 07:07:25  2427            calculated md5 sums = 2378020
STATISTIC 2016.02.04 07:07:25  2427                    forks total = 158172
STATISTIC 2016.02.04 07:07:25  2427                      forks md5 = 156734
STATISTIC 2016.02.04 07:07:25  2427                     forks copy = 1
STATISTIC 2016.02.04 07:07:25  2427                     forks lzop = 1170
STATISTIC 2016.02.04 07:07:25  2427                  sum of source = 350G (375655355323)
STATISTIC 2016.02.04 07:07:25  2427              sum of target all = 350G (375514069914)
STATISTIC 2016.02.04 07:07:25  2427              sum of target all = 99.96%
STATISTIC 2016.02.04 07:07:25  2427              sum of target new = 381M (399862140)
STATISTIC 2016.02.04 07:07:25  2427              sum of target new = 0.11%
STATISTIC 2016.02.04 07:07:25  2427             sum of md5ed files = 196G (210704369264)
STATISTIC 2016.02.04 07:07:25  2427             sum of md5ed files = 56.09%
STATISTIC 2016.02.04 07:07:25  2427     sum internal linked (copy) = 1.6M (1715765)
STATISTIC 2016.02.04 07:07:25  2427    sum internal linked (compr) = 4.1M (4334150)
STATISTIC 2016.02.04 07:07:25  2427          sum old linked (copy) = 130G (139874852149)
STATISTIC 2016.02.04 07:07:25  2427         sum old linked (compr) = 219G (235233305710)
STATISTIC 2016.02.04 07:07:25  2427           sum unchanged (copy) = 130G (139876567914)
STATISTIC 2016.02.04 07:07:25  2427          sum unchanged (compr) = 219G (235237639860)
STATISTIC 2016.02.04 07:07:25  2427                 sum new (copy) = 238M (249564876)
STATISTIC 2016.02.04 07:07:25  2427                sum new (compr) = 143M (150297264)
STATISTIC 2016.02.04 07:07:25  2427     sum new (compr), orig size = 275M (287888868)
STATISTIC 2016.02.04 07:07:25  2427                 sum new / orig = 74.40%
STATISTIC 2016.02.04 07:07:25  2427       size of md5CheckSum file =  76M (79326185)
STATISTIC 2016.02.04 07:07:25  2427     size of temporary db files = 0.0  (0)
STATISTIC 2016.02.04 07:07:25  2427            deleted old backups = 0
STATISTIC 2016.02.04 07:07:25  2427            deleted directories = 0
STATISTIC 2016.02.04 07:07:25  2427                  deleted files = 0
STATISTIC 2016.02.04 07:07:25  2427           (only) removed links = 0
STATISTIC 2016.02.04 07:07:25  2427 freed space in old directories = 0.0  (0)
STATISTIC 2016.02.04 07:07:25  2427       add. used space in files = 457M (479188325)
STATISTIC 2016.02.04 07:07:25  2427                backup duration = 7h6m4s
STATISTIC 2016.02.04 07:07:25  2427 over all files/sec (real time) = 105.57
STATISTIC 2016.02.04 07:07:25  2427  over all files/sec (CPU time) = 306.61
STATISTIC 2016.02.04 07:07:25  2427                      CPU usage = 34.43%

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code