mainstoreBackup - Support: sr #108652, Inconsistency not repaired by...

 
 

sr #108652: Inconsistency not repaired by --autorepair

Submitter:  Jonas <sanoj>
Submitted:  Sun 14 Sep 2014 08:30:04 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  4 - Important Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 01 Mar 2015 10:06:31 AM UTC, comment #8: 

From my point of view it's a wrong usage of that option. Man page or --help of storeBackupUpdateBackup.pl say:

top level directory of all backups (must exist)

Print a useful error message directlyi s pretty difficult because there's no special mark or so inside the backupDir. (A series name even may be '.').

So I finally limited the number of correction tries to 10 and changed the last output to:

ERROR     2015.03.01 11:04:05 22757 stopping - too many errors
ERROR     2015.03.01 11:04:05 22757 maybe you didn't specify the top level backupDir of all your backups?


Hope this helps.

Regards, Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Fri 27 Feb 2015 02:54:46 PM UTC, comment #7: 

I found the problem: storeBackupUpdateBackup does not work on a specific backup directory!

This script illustrates the problem. While you run storeBackupUpdateBackup on "b" it works but if you specify "b/default/2015....." it fails.

#!/bin/bash
mkdir storetest
cd storetest
rm -rf *
printf "\033c"
mkdir b s
cp /bin/ls s
cp /bin/ls s/ls1
storeBackup -s s -b b --lateLinks
find b -print | sort
storeBackupUpdateBackup -b b
find b -print | sort
storeBackup -s s -b b --lateLinks
storeBackupUpdateBackup -b b
find b -print | sort
storeBackup -s s -b b --lateLinks
storeBackupUpdateBackup -b b/default/`ls -tr b/default/ | tail -n1`

I am not sure if this is a bug or not, however if you now start storeBackupUpdateBackup with "-a" it totally messes up.


Helge <hellg>
Fri 27 Feb 2015 02:03:37 PM UTC, comment #6: 

I tried to simulate the behavior. For the data in sourceDir, I stored two times the same file, so there is an internal hardlink in each backup:

- mkdir s b

- cp /bin/ls s

- cp /bin/ls s/ls1

- storeBackup -g storetest.conf

- storetest.conf: sourceDir=s, backupDir=b, lateLinks=yes
- /opt/storeBackup/bin/storeBackup.pl -f storetest.conf
==> no error

-  find b -print | sort
b
b/default
b/default/2015.02.27_14.57.46
b/default/2015.02.27_14.57.46/ls1.bz2
b/default/2015.02.27_14.57.46/.md5CheckSums.bz2
b/default/2015.02.27_14.57.46/.md5CheckSums.info
b/default/2015.02.27_14.57.46/.storeBackupLinks
b/default/2015.02.27_14.57.46/.storeBackupLinks/backup.Finished
b/default/2015.02.27_14.57.46/.storeBackupLinks/linkFile.bz2
- /opt/storeBackup/bin/storeBackup.pl -f storetest.conf
==> no error

- find b -print | sort
b
b/default
b/default/2015.02.27_14.57.46
b/default/2015.02.27_14.57.46/ls1.bz2
b/default/2015.02.27_14.57.46/.md5CheckSums.bz2
b/default/2015.02.27_14.57.46/.md5CheckSums.info
b/default/2015.02.27_14.57.46/.storeBackupLinks
b/default/2015.02.27_14.57.46/.storeBackupLinks/backup.Finished
b/default/2015.02.27_14.57.46/.storeBackupLinks/linkFile.bz2
b/default/2015.02.27_14.57.46/.storeBackupLinks/linkFrom0
b/default/2015.02.27_14.58.39
b/default/2015.02.27_14.58.39/.md5CheckSums.bz2
b/default/2015.02.27_14.58.39/.md5CheckSums.info
b/default/2015.02.27_14.58.39/.storeBackupLinks
b/default/2015.02.27_14.58.39/.storeBackupLinks/backup.Finished
b/default/2015.02.27_14.58.39/.storeBackupLinks/linkFile.bz2
b/default/2015.02.27_14.58.39/.storeBackupLinks/linkTo

- /opt/storeBackup/bin/storeBackupUpdateBackup.pl -b b
==> no error

- find b -print | sort
b
b/default
b/default/2015.02.27_14.57.46
b/default/2015.02.27_14.57.46/ls1.bz2
b/default/2015.02.27_14.57.46/ls.bz2
b/default/2015.02.27_14.57.46/.md5CheckSums.bz2
b/default/2015.02.27_14.57.46/.md5CheckSums.info
b/default/2015.02.27_14.57.46/.storeBackupLinks
b/default/2015.02.27_14.57.46/.storeBackupLinks/backup.Finished
b/default/2015.02.27_14.58.39
b/default/2015.02.27_14.58.39/ls1.bz2
b/default/2015.02.27_14.58.39/ls.bz2
b/default/2015.02.27_14.58.39/.md5CheckSums.bz2
b/default/2015.02.27_14.58.39/.md5CheckSums.info
b/default/2015.02.27_14.58.39/.storeBackupLinks
b/default/2015.02.27_14.58.39/.storeBackupLinks/backup.Finished

------------------------

Can you send me a (not working) data set and complete configuration files and logs (as appended files) in an email?

Best Regards,
Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Thu 26 Feb 2015 09:17:44 AM UTC, comment #5: 

Hi,
I have the same problem here (latest version downloaded from server):
- storeBackup -g storetest.conf
- storetest.conf: sourceDir, backupDir changed, lateLinks=yes
- storeBackup -f storetest.conf
- storeBackupUpdateBackup -b backupDir/default/2015.02.26_09.43.19
- storeBackup -f storetest.conf
- storeBackupUpdateBackup -b backupDir/default/2015.02.26_09.43.39

BEGIN     2015.02.26 09:47:37  1412 checking references and backup copying in </mnt/temp/test_backup/default/2015.02.26_09.43.39>
VERSION   2015.02.26 09:47:37  1412 storeBackupUpdateBackup.pl, 3.5
INFO      2015.02.26 09:47:37  1412 removing old lock file of process <1370>
INFO      2015.02.26 09:47:37  1412 creating lock file </tmp/storeBackup.lock>
STATISTIC 2015.02.26 09:47:37  1412 found 0 backup series, 1 backups, 0 renamed backups
ERROR     2015.02.26 09:47:37  1412 1 directory </mnt/temp/test_backup/default/2015.02.26_09.43.19> has no linkFrom entry
ERROR     2015.02.26 09:47:37  1412 found 1 inconsistencies, please repair and check again

However:
cat /mnt/temp/test_backup/default/2015.02.26_09.43.19/.storeBackupLinks/linkFrom0
../2015.02.26_09.43.39

If I now use storeBackupUpdateBackup -a I get thousands of "linkFromXXXX" entries in the above mentioned directory. I already tried to debug the behaviour but I can't find the code where it searches for the "linkFrom" file.

Do I use the lateLinks option wrong or is there any way to help debug the problem? If you need any other debug info just let me know.

Best regards,
Helge.

Helge <hellg>
Tue 16 Sep 2014 02:26:58 PM UTC, comment #4: 

Hi,

sorry, i couldn't create a minimal example for reproduction... It always worked. However, I deleted the corrupted backup. All other backups are fine...
Cheers

Jonas <sanoj>
Tue 16 Sep 2014 09:53:32 AM UTC, comment #3: 


> The autorepair creates literally 1000s of LinkFrom files.


The linkFrom files just tell the storeBackup scripts that there is at least one latelink in linkFile.bz2 pointing to the listed series.

If there are 1000s of linkFrom files, then somethings definitely wrong. Does it is possible to create something reproducible so I understand what really happens?

I think you should save the new backup (mv outside the backupDir) and also those 1000s of linkFrom files to somewhere. Are other backups affected (look for linkFrom* and linkTo files)? If there is a linkFile.bz2, then that particular backup was created with lateLinks and is not finished with storeBackupUpdateBackup.pl. If you clean everything that way, storeBackupUpdateBackup.pl should run without doing anything with the backups. Then start a new backup and see what happens.

Regards.

Heinz-Josef Claes <hjclaes>
Group administrator
Tue 16 Sep 2014 08:57:57 AM UTC, comment #2: 

HI,

no, I'm using lateLinks for quite some time and it did work without problems until now. I don't know what changed...

The autorepair creates literally 1000s of LinkFrom files. The linkFile.bz2 is missing, and that's probably the problem. I think the backup is actually finished (even the lateLinks) but somehow storebackup still tries to create the links. As the linkFile.bz2 is missing the error occurs.

I will just delete the currupted backup. However the autorepair should check if the according linkTo file really doesn't exist before it creates endless copies of the same file.

Thanks,
Jonas

Jonas <sanoj>
Mon 15 Sep 2014 01:56:58 PM UTC, comment #1: 

Hi,

do you use lateLinks for the very first time, do these messages happen after some time of using lateLinks or did you do any manipulations manually?

To understand what's happening, see
"7.9 Special files generated and used by storeBackup"

Setup a test environment and run a first backup:

 mkcd /tmp/x
 cd /tmp/x
 mkdir s b
 cp /bin/ls s
 /opt/storeBackup/bin/storeBackup.pl -s s -b b

look into the special directory .storeBackupLinks:

 ls -l b/default/2014.09.15_15.49.41/.storeBackupLinks/
insgesamt 0
-rw-r--r-- 1 hjc hjc 0 Sep 15 15:49 backup.Finished

-> nothing special in it.

now run a backup with lateLinks

 /opt/storeBackup/bin/storeBackup.pl -s s -b b --lateLinks

now look into that directory again:

 ls -l b/default/2014.*/.storeBackupLinks/
b/default/2014.09.15_15.49.41/.storeBackupLinks/:
insgesamt 4
-rw-r--r-- 1 hjc hjc  0 Sep 15 15:49 backup.Finished
-rw------- 1 hjc hjc 23 Sep 15 15:51 linkFrom0

b/default/2014.09.15_15.51.44/.storeBackupLinks/:
insgesamt 8
-rw-r--r-- 1 hjc hjc   0 Sep 15 15:51 backup.Finished
-rw------- 1 hjc hjc 201 Sep 15 15:51 linkFile.bz2
-rw------- 1 hjc hjc  23 Sep 15 15:51 linkTo

look into the (ascii) files in .storeBackupLinks to understand how they are used. (You assume also read that chapter 7.9)

If you have success in some way or not or if my explanation is not understandable, please report.

Regards, Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Sun 14 Sep 2014 08:30:04 PM UTC, original submission:  

Hi,

I'm running storebackup with latelinks. When running the UpdateBackup script I get the follwing error:

BEGIN     2014.09.14 21:23:53   381 checking references and backup copying in <2014.08.28_23.41.47>
VERSION   2014.09.14 21:23:53   381 storeBackupUpdateBackup.pl, 3.5
INFO      2014.09.14 21:23:53   381 removing old lock file of process <365>
INFO      2014.09.14 21:23:53   381 creating lock file </tmp/storeBackup.lock>
STATISTIC 2014.09.14 21:23:53   381 found 0 backup series, 1 backups, 0 renamed backups
ERROR     2014.09.14 21:23:53   381 1 directory <<BACKUPDIR><SERIES1>/2014.07.10_15.29.04> has no linkFrom entry
ERROR     2014.09.14 21:23:53   381 1 directory <<BACKUPDIR><SERIES1>/2014.08.25_18.12.53> has no linkFrom entry
ERROR     2014.09.14 21:23:53   381 1 directory <<BACKUPDIR><SERIES2>/2013.10.22_13.05.25> has no linkFrom entry
ERROR     2014.09.14 21:23:53   381 1 directory <<BACKUPDIR><SERIES2>/2014.04.05_12.17.43> has no linkFrom entry
ERROR     2014.09.14 21:23:53   381 1 directory <<BACKUPDIR><SERIES2>/2014.05.27_23.40.36> has no linkFrom entry
ERROR     2014.09.14 21:23:53   381 1 directory <<BACKUPDIR><SERIES2>/2014.06.22_18.45.56> has no linkFrom entry
ERROR     2014.09.14 21:23:53   381 1 directory <<BACKUPDIR><SERIES3>/2014.07.27_11.43.02> has no linkFrom entry
ERROR     2014.09.14 21:23:53   381 2 directory <<BACKUPDIR><SERIES1>/2014.09.14_00.48.29> has no entry
ERROR     2014.09.14 21:23:53   381 in linkTo to <<BACKUPDIR><SERIES1>/2014.08.28_23.41.47> but
ERROR     2014.09.14 21:23:53   381 <<BACKUPDIR><SERIES1>/2014.08.28_23.41.47> has linkFrom to <<BACKUPDIR><SERIES1>/2014.09.14_00.48.29>
ERROR     2014.09.14 21:23:53   381 found 8 inconsistencies, please repair and check again


When I run it with the autorepair option it seems to be in an endless loop saying


ERROR     2014.09.14 21:28:34   556 ----- repeating consistency check -----
STATISTIC 2014.09.14 21:28:34   556 found 0 backup series, 1 backups, 0 renamed backups
ERROR     2014.09.14 21:28:34   556 1 directory <<BACKUPDIR><SERIES1>/2014.07.10_15.29.04> has no linkFrom entry
WARNING   2014.09.14 21:28:34   556 autorepair: wrote linkFrom from <<BACKUPDIR><SERIES1>/2014.07.10_15.29.04> to <<BACKUPDIR><SERIES1>/2014.08.28_23.41.47>
ERROR     2014.09.14 21:28:34   556


1000s of times. How can I repair this error (manually or automatically...?)

Thanks,
Jonas

Jonas <sanoj>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hellg (Posted a comment)
  • -email is unavailable- added by hjclaes (Posted a comment)
  • -email is unavailable- added by sanoj (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code