bugstoreBackup - Bugs: bug #40255, backup big file failed

 
 

bug #40255: backup big file failed

Submitted by:  None
Submitted on:  Sun 13 Oct 2013 03:30:34 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: None
Privacy: PublicAssigned to: None
Open/Closed: Open

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Fri 18 Oct 2013 07:37:22 AM UTC, comment #6:

Did you double check the new backup with version 3.4.2? Does it show any faults?

Heinz-Josef Claes <hjclaes>
Project Administrator
Thu 17 Oct 2013 07:08:08 PM UTC, comment #5:

storeBackup.pl --print <YOUR_OPTIONS>
I made it into shell:
storeBackup.pl --sourceDir $source --backupDir $target --logFile $global_log -f $global_conf --keepAll ${day}d

The $source is sourcedir,$target is backupdir,$day is 3

I run backup in a clean environment which the backup dir is empty.

I find the .md5CheckSums.info into the backup dir.

The content is as blow:
version=1.3
date=2013.10.14 04.43.33
sourceDir='/home/mysql/mysql/var'
followLinks=0
compress='gzip'
uncompress='gzip -d'
postfix='.gz'
comprRule='$size > 1024 and' 'not $file =~ /\.frm\Z/i'
exceptDirs=
includeDirs=
exceptRule=
includeRule=
writeExcludeLog=no
exceptTypes=
archiveTypes=
specialTypeArchiver=cpio
checkBlocksRule0=
checkBlocksBS0=
checkBlocksCompr0=
checkBlocksRead0=
checkBlocksRule1=
checkBlocksBS1=
checkBlocksCompr1=
checkBlocksRead1=
checkBlocksRule2=
checkBlocksBS2=
checkBlocksCompr2=
checkBlocksRead2=
checkBlocksRule3=
checkBlocksBS3=
checkBlocksCompr3=
checkBlocksRead3=
checkBlocksRule4=
checkBlocksBS4=
checkBlocksCompr4=
checkBlocksRead4=
checkBlocksRule5='$size >= &::SIZE("3G") and $file =~ /\5C.ibd\5CZ/'
checkBlocksBS5=104857600
checkBlocksCompr5=yes
checkBlocksRead5=
preservePerms=yes
lateLinks=no
lateCompress=no
cpIsGnu=no
logInBackupDir=no
compressLogInBackupDir=no
logInBackupDirFileName='.storeBackup.log'
linkToRecent=

Thank you for your patience.

zhaochengyong <imzcy1987>
Tue 15 Oct 2013 01:55:53 PM UTC, comment #4:

I made some tests with a 20GB kvm image with gzip.
The very first backup was ok (tested with zcat | md5sum) and storeBackupCheckBackup.pl
The a changed the image by booting the OS running on it and making updates to that Linux OS. (So there were a lot of changes.)
Also - no problem.
(I didn't use lateLinks.)

Can you post the output of
storeBackup.pl --print <YOUR_OPTIONS> ?

Please use the most actual version for your tests.

Can you run your backup in a clean environment, so no references to old backups (which might be corrupted for whatever reasons)?
To do this into the same backup folder, you can set series to eg. "mySeries". Then set in the configuration file:
otherBackupSeries=0:mySeries
This tells storeBackup.pl to link to series mySeries only.

The errors reported probably are from not initialized values for statistics only.

Heinz-Josef Claes <hjclaes>
Project Administrator
Tue 15 Oct 2013 07:40:57 AM UTC, comment #3:

I tested again and found some error log.
MayBe ti can make some help.

Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7446.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7488.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7418.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 7427.
Use of uninitialized value in addition (+) at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 6906.
Use of uninitialized value in pack at /home/mysql/opbin/hbak/crontab_673/storeBackup/bin/storeBackup.pl line 4321.

zhaochengyong <imzcy1987>
Mon 14 Oct 2013 08:34:07 AM UTC, comment #2:

The disk is not full.

Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p2 8.7G 3.2G 5.6G 37% /
/dev/cciss/c0d1p1 4.8T 560G 4.0T 13% /home
/dev/cciss/c0d1p2 5.3T 532G 4.5T 11% /home/data

I test the tool again. I found the block file compressed with gzip is diffient from source file.

When I uncompressed the block file list,some files' md5 is diffient from .md5BlockCheckSums.bz2.

I sumbit a picture which show diffents between uncompressed file list and .md5BlockCheckSums.bz2.

(file #29373)

zhaochengyong <imzcy1987>
Sun 13 Oct 2013 04:03:27 PM UTC, comment #1:

Hi,

Does any of the affected filesystems is full? Try:
$ df -h

storeBackup didn't report a full filesystem properly. This was fixed in 3.4.2

Heinz-Josef Claes <hjclaes>
Project Administrator
Sun 13 Oct 2013 03:30:34 PM UTC, original submission:

When I backed up a directory,in which there is a large file, I choose the way using block file storage and compression.
But I found that when the file is large. The file recovered from the backup is much smaller than the source files, backup is problematic.
And in the directory named by the file name, there is no file(.Md5BlockCheckSums.bz2),this compression block file list.
I found nothing abnormal in the log, the problem is this big file user_message. Ibd (49 g)
The backup log is as blow,please help me to check the reasion.Thanks very much.

BEGIN 2013.10.12 00:18:50 29693 backing up directory </home/data/mysql/var> to </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup/default>
VERSION 2013.10.12 00:18:50 29693 storeBackup.pl, 3.4
INFO 2013.10.12 00:18:50 29693 setting ARG_MAX to 63488 (Linux)
INFO 2013.10.12 00:18:50 29693 comprRule = $size > 1024 and not $file =~ /\.frm\Z/i
INFO 2013.10.12 00:18:50 29693 scanning directory </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup> for existing backups
INFO 2013.10.12 00:18:50 29693 scanning directory </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup/default> for existing backups
STATISTIC 2013.10.12 00:18:50 29693 found 1 backup series, 3 backups, 0 renamed backups
INFO 2013.10.12 00:18:50 29693 consistency check finished successfully
INFO 2013.10.12 00:18:50 29693 found no references to backups from lateLinks that need storeBackupUpdateBackup run
INFO 2013.10.12 00:18:50 29693 checkBlocksRule = <$size >= &::SIZE("3G") and $file =~ /\.ibd\Z/ ; >
INFO 2013.10.12 00:18:50 29693 creating lock file <./storeBackup.lock>
INFO 2013.10.12 00:18:50 29693 analysis of old Backups in </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup/default>:
INFO 2013.10.12 00:18:50 29693 Wed 2013.10.09_01.32.02 (2d22h): keepMinNumber4, keepWeekDays(4d)
INFO 2013.10.12 00:18:50 29693 Thu 2013.10.10_00.49.42 (1d23h): keepMinNumber3, keepWeekDays(4d)
INFO 2013.10.12 00:18:50 29693 Fri 2013.10.11_02.42.35 (0d21h): keepMinNumber2, keepWeekDays(4d)
INFO 2013.10.12 00:18:50 29693 Sat 2013.10.12_00.18.50 (0d0h): keepMinNumber1, keepWeekDays(4d)
INFO 2013.10.12 00:18:50 29693 0 entries in dbm files
INFO 2013.10.12 00:18:50 29693 0 entries in dbm block files
WARNING 2013.10.12 00:20:03 29693 file </home/data/mysql/var/relay-log.004041> changed during backup
WARNING 2013.10.12 00:20:03 29693 file </home/data/mysql/var/master.info> changed during backup
INFO 2013.10.12 00:20:14 29693 saving blocked file <notes/posts.ibd> (3.1G)
INFO 2013.10.12 00:21:38 29693 saving blocked file <user/user_message.ibd> ( 49G)
INFO 2013.10.12 00:24:33 29693 saving blocked file <user/posts_draft.ibd> (3.1G)
INFO 2013.10.12 00:25:18 29693 saving blocked file <user/user_ext.ibd> (4.3G)
INFO 2013.10.12 00:26:14 29693 saving blocked file <user/user.ibd> (4.6G)
INFO 2013.10.12 00:28:38 29693 saving blocked file <scene/version.ibd> (9.0G)
INFO 2013.10.12 00:31:10 29693 setting atime, mtime of directories ...
STATISTIC 2013.10.12 00:31:10 29693 [sec] | user| system
STATISTIC 2013.10.12 00:31:10 29693 -------+----------+----------
STATISTIC 2013.10.12 00:31:10 29693 process| 0.99| 0.41
STATISTIC 2013.10.12 00:31:10 29693 childs | 3130.89| 224.47
STATISTIC 2013.10.12 00:31:10 29693 -------+----------+----------
STATISTIC 2013.10.12 00:31:10 29693 sum | 3131.88| 224.88 => 3356.76 (55m57s)
STATISTIC 2013.10.12 00:31:10 29693 directories = 15
STATISTIC 2013.10.12 00:31:10 29693 files = 1363
STATISTIC 2013.10.12 00:31:10 29693 symbolic links = 0
STATISTIC 2013.10.12 00:31:10 29693 late links = 1
STATISTIC 2013.10.12 00:31:10 29693 named pipes = 0
STATISTIC 2013.10.12 00:31:10 29693 sockets = 0
STATISTIC 2013.10.12 00:31:10 29693 block devices = 0
STATISTIC 2013.10.12 00:31:10 29693 character devices = 0
STATISTIC 2013.10.12 00:31:10 29693 new internal linked files = 24
STATISTIC 2013.10.12 00:31:10 29693 old linked files = 0
STATISTIC 2013.10.12 00:31:10 29693 unchanged files = 24
STATISTIC 2013.10.12 00:31:10 29693 copied files = 1115
STATISTIC 2013.10.12 00:31:10 29693 compressed files = 224
STATISTIC 2013.10.12 00:31:10 29693 blocked files = 6
STATISTIC 2013.10.12 00:31:10 29693 excluded files because rule = 0 (0.0 )
STATISTIC 2013.10.12 00:31:10 29693 included files because rule = 0 (0.0 )
STATISTIC 2013.10.12 00:31:10 29693 max size of copy queue = 1
STATISTIC 2013.10.12 00:31:10 29693 max size of compression queue = 38
STATISTIC 2013.10.12 00:31:10 29693 calculated md5 sums = 2946
STATISTIC 2013.10.12 00:31:10 29693 forks total = 496
STATISTIC 2013.10.12 00:31:10 29693 forks md5 = 21
STATISTIC 2013.10.12 00:31:10 29693 forks copy = 1
STATISTIC 2013.10.12 00:31:10 29693 forks gzip = 474
STATISTIC 2013.10.12 00:31:10 29693 sum of source = 40G (43156647341)
STATISTIC 2013.10.12 00:31:10 29693 sum of target all = 8.6G (9266277743)
STATISTIC 2013.10.12 00:31:10 29693 sum of target all = 21.47%
STATISTIC 2013.10.12 00:31:10 29693 sum of target new = 8.6G (9266259359)
STATISTIC 2013.10.12 00:31:10 29693 sum of target new = 21.47%
STATISTIC 2013.10.12 00:31:10 29693 sum of md5ed files = 40G (43154743966)
STATISTIC 2013.10.12 00:31:10 29693 sum of md5ed files = 100.00%
STATISTIC 2013.10.12 00:31:10 29693 sum internal linked (copy) = 18k (18384)
STATISTIC 2013.10.12 00:31:10 29693 sum internal linked (compr) = 0.0 (0)
STATISTIC 2013.10.12 00:31:10 29693 sum old linked (copy) = 0.0 (0)
STATISTIC 2013.10.12 00:31:10 29693 sum old linked (compr) = 0.0 (0)
STATISTIC 2013.10.12 00:31:10 29693 sum unchanged (copy) = 18k (18384)
STATISTIC 2013.10.12 00:31:10 29693 sum unchanged (compr) = 0.0 (0)
STATISTIC 2013.10.12 00:31:10 29693 sum new (copy) = 2.3M (2434440)
STATISTIC 2013.10.12 00:31:10 29693 sum new (compr) = 8.6G (9263824919)
STATISTIC 2013.10.12 00:31:10 29693 sum new (compr), orig size = 40G (43154195717)
STATISTIC 2013.10.12 00:31:10 29693 sum new / orig = 21.47%
STATISTIC 2013.10.12 00:31:10 29693 size of md5CheckSum file = 47k (48528)
STATISTIC 2013.10.12 00:31:10 29693 size of temporary db files = 0.0 (0)
STATISTIC 2013.10.12 00:31:10 29693 deleted old backups = 0
STATISTIC 2013.10.12 00:31:10 29693 deleted directories = 0
STATISTIC 2013.10.12 00:31:10 29693 deleted files = 0
STATISTIC 2013.10.12 00:31:10 29693 (only) removed links = 0
STATISTIC 2013.10.12 00:31:10 29693 freed space in old directories = 0.0 (0)
STATISTIC 2013.10.12 00:31:10 29693 add. used space in files = 8.6G (9266307887)
STATISTIC 2013.10.12 00:31:10 29693 backup duration = 12m20s
STATISTIC 2013.10.12 00:31:10 29693 over all files/sec (real time) = 1.84
STATISTIC 2013.10.12 00:31:10 29693 over all files/sec (CPU time) = 0.41
STATISTIC 2013.10.12 00:31:10 29693 CPU usage = 453.62%
INFO 2013.10.12 00:31:10 29693 removing lock file <./storeBackup.lock>
WARNING 2013.10.12 00:31:11 29693 -- 2 WARNINGS OCCURRED DURING THE BACKUP! --
INFO 2013.10.12 00:31:11 29693 syncing ...
END 2013.10.12 00:31:11 29693 backing up directory </home/data/mysql/var> to </home/data/opdir/mysqlbackup/mysqldata_mysql/coldbackup/default/2013.10.12_00.18.5
0>

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #29373:  diff.png added by imzcy1987 (171KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by imzcy1987 (Updated the item)
  • -unavailable- added by hjclaes (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 14 Oct 2013 08:34:07 AM UTCimzcy1987Attached File-=>Added diff.png, #29373

    Back to the top


    Powered by Savane 3.1-cleanup1