bugstoreBackup - Bugs: bug #40255, backup big file failed

 
 

bug #40255: backup big file failed

Submitter:  None
Submitted:  Sun 13 Oct 2013 03:30:34 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 18 Oct 2013 07:37:22 AM UTC, comment #6: 

Did you double check the new backup with version 3.4.2? Does it show any faults?

 Heinz-Josef Claes <hjclaes>
Group administrator
Thu 17 Oct 2013 07:08:08 PM UTC, comment #5: 

storeBackup.pl --print <YOUR_OPTIONS>
I made it into shell:
storeBackup.pl --sourceDir $source  --backupDir $target --logFile $global_log -f $global_conf --keepAll ${day}d

The $source is sourcedir,$target is backupdir,$day is 3

I run backup in a clean environment which the backup dir is empty.

I find the .md5CheckSums.info into the backup dir.

The content is as blow:
version=1.3
date=2013.10.14 04.43.33
sourceDir='/home/mysql/mysql/var'
followLinks=0
compress='gzip'
uncompress='gzip -d'
postfix='.gz'
comprRule='$size > 1024 and' 'not $file =~ /\.frm\Z/i'
exceptDirs=
includeDirs=
exceptRule=
includeRule=
writeExcludeLog=no
exceptTypes=
archiveTypes=
specialTypeArchiver=cpio
checkBlocksRule0=
checkBlocksBS0=
checkBlocksCompr0=
checkBlocksRead0=
checkBlocksRule1=
checkBlocksBS1=
checkBlocksCompr1=
checkBlocksRead1=
checkBlocksRule2=
checkBlocksBS2=
checkBlocksCompr2=
checkBlocksRead2=
checkBlocksRule3=
checkBlocksBS3=
checkBlocksCompr3=
checkBlocksRead3=
checkBlocksRule4=
checkBlocksBS4=
checkBlocksCompr4=
checkBlocksRead4=
checkBlocksRule5='$size >= &::SIZE("3G") and $file =~ /\5C.ibd\5CZ/'
checkBlocksBS5=104857600
checkBlocksCompr5=yes
checkBlocksRead5=
preservePerms=yes
lateLinks=no
lateCompress=no
cpIsGnu=no
logInBackupDir=no
compressLogInBackupDir=no
logInBackupDirFileName='.storeBackup.log'
linkToRecent=


Thank you for your patience.

 zhaochengyong <imzcy1987>
Tue 15 Oct 2013 01:55:53 PM UTC, comment #4: 

I made some tests with a 20GB kvm image with gzip.
The very first backup was ok (tested with zcat | md5sum) and storeBackupCheckBackup.pl
The a changed the image by booting the OS running on it and making updates to that Linux OS. (So there were a lot of changes.)
Also - no problem.
(I didn't use lateLinks.)

Can you post the output of
storeBackup.pl --print <YOUR_OPTIONS> ?

Please use the most actual version for your tests.

Can you run your backup in a clean environment, so no references to old backups (which might be corrupted for whatever reasons)?
To do this into the same backup folder, you can set series to eg. "mySeries". Then set in the configuration file:
otherBackupSeries=0:mySeries
This tells storeBackup.pl to link to series mySeries only.

The errors reported probably are from not initialized values for statistics only.

 Heinz-Josef Claes <hjclaes>
Group administrator
Tue 15 Oct 2013 07:40:57 AM UTC, comment #3: 

I tested again and found some error log.
MayBe ti can make some help.

Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.

 zhaochengyong <imzcy1987>
Mon 14 Oct 2013 08:34:07 AM UTC, comment #2: 

The disk is not full.

Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p2     8.7G  3.2G  5.6G  37% /
/dev/cciss/c0d1p1     4.8T  560G  4.0T  13% /home
/dev/cciss/c0d1p2     5.3T  532G  4.5T  11% /home/data

I test the tool again. I found the block file compressed with gzip is diffient from source file.

When I uncompressed the block file list,some files' md5 is diffient from .md5BlockCheckSums.bz2.

I sumbit a picture which show diffents between uncompressed file list and .md5BlockCheckSums.bz2.






zhaochengyong <imzcy1987>
Sun 13 Oct 2013 04:03:27 PM UTC, comment #1: 

Hi,

Does any of the affected filesystems is full? Try:
$ df -h

storeBackup didn't report a full filesystem properly. This was fixed in 3.4.2

Heinz-Josef Claes <hjclaes>
Group administrator
Sun 13 Oct 2013 03:30:34 PM UTC, original submission:  

When I backed up a directory,in which there is a large file, I choose the way using block file storage and compression.
But I found that when the file is large. The file recovered from the backup is much smaller than the source files, backup is problematic.
And in the directory named by the file name, there is no file(.Md5BlockCheckSums.bz2),this compression block file list.
I found nothing abnormal in the log, the problem is this big file user_message. Ibd (49 g)
The backup log is as blow,please help me to check the reasion.Thanks very much.


BEGIN     2013.10.12 00:18:50 29693 backing up directory </home/data/mysql/var> to </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup/default>
VERSION   2013.10.12 00:18:50 29693 storeBackup.pl, 3.4
INFO      2013.10.12 00:18:50 29693 setting ARG_MAX to 63488 (Linux)
INFO      2013.10.12 00:18:50 29693 comprRule = $size > 1024 and not $file =~ /\.frm\Z/i
INFO      2013.10.12 00:18:50 29693 scanning directory </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup> for existing backups
INFO      2013.10.12 00:18:50 29693 scanning directory </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup/default> for existing backups
STATISTIC 2013.10.12 00:18:50 29693 found 1 backup series, 3 backups, 0 renamed backups
INFO      2013.10.12 00:18:50 29693 consistency check finished successfully
INFO      2013.10.12 00:18:50 29693 found no references to backups from lateLinks that need storeBackupUpdateBackup run
INFO      2013.10.12 00:18:50 29693 checkBlocksRule = <$size >= &::SIZE("3G") and $file =~ /\.ibd\Z/ ; >
INFO      2013.10.12 00:18:50 29693 creating lock file <./storeBackup.lock>
INFO      2013.10.12 00:18:50 29693 analysis of old Backups in </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup/default>:
INFO      2013.10.12 00:18:50 29693    Wed 2013.10.09_01.32.02 (2d22h): keepMinNumber4, keepWeekDays(4d)
INFO      2013.10.12 00:18:50 29693    Thu 2013.10.10_00.49.42 (1d23h): keepMinNumber3, keepWeekDays(4d)
INFO      2013.10.12 00:18:50 29693    Fri 2013.10.11_02.42.35 (0d21h): keepMinNumber2, keepWeekDays(4d)
INFO      2013.10.12 00:18:50 29693    Sat 2013.10.12_00.18.50 (0d0h): keepMinNumber1, keepWeekDays(4d)
INFO      2013.10.12 00:18:50 29693 0 entries in dbm files
INFO      2013.10.12 00:18:50 29693 0 entries in dbm block files
WARNING   2013.10.12 00:20:03 29693 file </home/data/mysql/var/relay-log.004041> changed during backup
WARNING   2013.10.12 00:20:03 29693 file </home/data/mysql/var/master.info> changed during backup
INFO      2013.10.12 00:20:14 29693 saving blocked file <notes/posts.ibd> (3.1G)
INFO      2013.10.12 00:21:38 29693 saving blocked file <user/user_message.ibd> ( 49G)
INFO      2013.10.12 00:24:33 29693 saving blocked file <user/posts_draft.ibd> (3.1G)
INFO      2013.10.12 00:25:18 29693 saving blocked file <user/user_ext.ibd> (4.3G)
INFO      2013.10.12 00:26:14 29693 saving blocked file <user/user.ibd> (4.6G)
INFO      2013.10.12 00:28:38 29693 saving blocked file <scene/version.ibd> (9.0G)
INFO      2013.10.12 00:31:10 29693 setting atime, mtime of directories ...
STATISTIC 2013.10.12 00:31:10 29693  [sec] |      user|    system
STATISTIC 2013.10.12 00:31:10 29693 -------+----------+----------
STATISTIC 2013.10.12 00:31:10 29693 process|      0.99|      0.41
STATISTIC 2013.10.12 00:31:10 29693 childs |   3130.89|    224.47
STATISTIC 2013.10.12 00:31:10 29693 -------+----------+----------
STATISTIC 2013.10.12 00:31:10 29693 sum    |   3131.88|    224.88 => 3356.76 (55m57s)
STATISTIC 2013.10.12 00:31:10 29693                    directories = 15
STATISTIC 2013.10.12 00:31:10 29693                          files = 1363
STATISTIC 2013.10.12 00:31:10 29693                 symbolic links = 0
STATISTIC 2013.10.12 00:31:10 29693                     late links = 1
STATISTIC 2013.10.12 00:31:10 29693                    named pipes = 0
STATISTIC 2013.10.12 00:31:10 29693                        sockets = 0
STATISTIC 2013.10.12 00:31:10 29693                  block devices = 0
STATISTIC 2013.10.12 00:31:10 29693              character devices = 0
STATISTIC 2013.10.12 00:31:10 29693      new internal linked files = 24
STATISTIC 2013.10.12 00:31:10 29693               old linked files = 0
STATISTIC 2013.10.12 00:31:10 29693                unchanged files = 24
STATISTIC 2013.10.12 00:31:10 29693                   copied files = 1115
STATISTIC 2013.10.12 00:31:10 29693               compressed files = 224
STATISTIC 2013.10.12 00:31:10 29693                  blocked files = 6
STATISTIC 2013.10.12 00:31:10 29693    excluded files because rule = 0 (0.0 )
STATISTIC 2013.10.12 00:31:10 29693    included files because rule = 0 (0.0 )
STATISTIC 2013.10.12 00:31:10 29693         max size of copy queue = 1
STATISTIC 2013.10.12 00:31:10 29693  max size of compression queue = 38
STATISTIC 2013.10.12 00:31:10 29693            calculated md5 sums = 2946
STATISTIC 2013.10.12 00:31:10 29693                    forks total = 496
STATISTIC 2013.10.12 00:31:10 29693                      forks md5 = 21
STATISTIC 2013.10.12 00:31:10 29693                     forks copy = 1
STATISTIC 2013.10.12 00:31:10 29693                     forks gzip = 474
STATISTIC 2013.10.12 00:31:10 29693                  sum of source =  40G (43156647341)
STATISTIC 2013.10.12 00:31:10 29693              sum of target all = 8.6G (9266277743)
STATISTIC 2013.10.12 00:31:10 29693              sum of target all = 21.47%
STATISTIC 2013.10.12 00:31:10 29693              sum of target new = 8.6G (9266259359)
STATISTIC 2013.10.12 00:31:10 29693              sum of target new = 21.47%
STATISTIC 2013.10.12 00:31:10 29693             sum of md5ed files =  40G (43154743966)
STATISTIC 2013.10.12 00:31:10 29693             sum of md5ed files = 100.00%
STATISTIC 2013.10.12 00:31:10 29693     sum internal linked (copy) =  18k (18384)
STATISTIC 2013.10.12 00:31:10 29693    sum internal linked (compr) = 0.0  (0)
STATISTIC 2013.10.12 00:31:10 29693          sum old linked (copy) = 0.0  (0)
STATISTIC 2013.10.12 00:31:10 29693         sum old linked (compr) = 0.0  (0)
STATISTIC 2013.10.12 00:31:10 29693           sum unchanged (copy) =  18k (18384)
STATISTIC 2013.10.12 00:31:10 29693          sum unchanged (compr) = 0.0  (0)
STATISTIC 2013.10.12 00:31:10 29693                 sum new (copy) = 2.3M (2434440)
STATISTIC 2013.10.12 00:31:10 29693                sum new (compr) = 8.6G (9263824919)
STATISTIC 2013.10.12 00:31:10 29693     sum new (compr), orig size =  40G (43154195717)
STATISTIC 2013.10.12 00:31:10 29693                 sum new / orig = 21.47%
STATISTIC 2013.10.12 00:31:10 29693       size of md5CheckSum file =  47k (48528)
STATISTIC 2013.10.12 00:31:10 29693     size of temporary db files = 0.0  (0)
STATISTIC 2013.10.12 00:31:10 29693            deleted old backups = 0
STATISTIC 2013.10.12 00:31:10 29693            deleted directories = 0
STATISTIC 2013.10.12 00:31:10 29693                  deleted files = 0
STATISTIC 2013.10.12 00:31:10 29693           (only) removed links = 0
STATISTIC 2013.10.12 00:31:10 29693 freed space in old directories = 0.0  (0)
STATISTIC 2013.10.12 00:31:10 29693       add. used space in files = 8.6G (9266307887)
STATISTIC 2013.10.12 00:31:10 29693                backup duration = 12m20s
STATISTIC 2013.10.12 00:31:10 29693 over all files/sec (real time) = 1.84
STATISTIC 2013.10.12 00:31:10 29693  over all files/sec (CPU time) = 0.41
STATISTIC 2013.10.12 00:31:10 29693                      CPU usage = 453.62%
INFO      2013.10.12 00:31:10 29693 removing lock file <./storeBackup.lock>
WARNING   2013.10.12 00:31:11 29693 -- 2 WARNINGS OCCURRED DURING THE BACKUP! --
INFO      2013.10.12 00:31:11 29693 syncing ...
END       2013.10.12 00:31:11 29693 backing up directory </home/data/mysql/var> to </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup/default/2013.10.12_00.18.5
0>

 Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #29373:  diff.png added by imzcy1987 (171KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by imzcy1987 (Updated the item)
  • -email is unavailable- added by hjclaes (Posted a comment)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-10-14 imzcy1987 Attached File- Added diff.png, #29373

    Back to the top

    Copyright © 2025 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-04e1.
    Corresponding source code