bugrdiff-backup - Bugs: bug #26848, Restoring hardlinked files:...

 
 

bug #26848: Restoring hardlinked files: "Hash for foo missing, cannot check"

Submitter:  Joe Steele <j_steele>
Submitted:  Fri 19 Jun 2009 09:30:56 PM UTC
Votes: 105
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Duplicate
Privacy:  Public Assigned to:  None
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 24 Feb 2020 10:28:55 AM UTC, comment #5: 

Closing as now handled in GitHub (see previous comment).

Eric L. <ericzolf>
Group administrator
Sun 19 Jan 2020 03:50:07 AM UTC, comment #4: 

I have now submitted the attached patches as part of this GitGub pull request:

https://github.com/rdiff-backup/rdiff-backup/pull/240

Joe Steele <j_steele>
Tue 17 Aug 2010 08:35:17 PM UTC, comment #3: 

More problems with hardlinked files...

Since the time of my original bug submission on 6/19/09, I have encountered a different test case that was not fixed by the original patch that I had submitted.  New patches are attached for Hardlink.py and compare.py that fix the problems demonstrated in the following example (patches are against version 1.2.8).

Create 2 files that are hardlinked to each other, back them up, then move the 'first' file so that it comes after the second file, then back them up again.  The SHA1 digest is lost (no longer in the mirror metadata).  Errors are generated when trying to restore the backup, and when verifying the backup, and when comparing the hash of the backup:

$ mkdir folder
$ touch folder/a
$ link folder/a folder/b
$ rdiff-backup folder backup
$ mv folder/a folder/c
$ sleep 1
$ rdiff-backup folder backup
$ mkdir restore
$ rdiff-backup -r now backup restore
Hash for b missing, cannot check

$ rdiff-backup --verify backup
Warning: Cannot find SHA1 digest for file b,
perhaps because this feature was added in v1.1.1
Warning: Cannot find SHA1 digest for file c,
perhaps because this feature was added in v1.1.1
Every file verified successfully.

$ rdiff-backup --compare-hash folder backup
Warning: Metadata file has no digest for b, unable to compare.
Warning: Metadata file has no digest for c, unable to compare.
No changes found.  Directory matches archive data.

(No doubt bugs #20925 and #27391 are related to the above problems.)

(file #21244, file #21245)

Joe Steele <j_steele>
Thu 17 Jun 2010 07:49:07 PM UTC, comment #2: 

Hello,

This bug report is more than a year old and and no one is paying attention to it. I think this is very serious since it affects data integrity.


I can reproduce this bug on my servers and I think its very serious. If you use rdiff-backup to backup your root filesystem you can end easily end with a bad backup regarding to hardlinked files because rdiff-backup will completely mess with them


This patch seems to solve it as I saw.

carlos lopez <carloslp>
Wed 25 Nov 2009 10:50:47 PM UTC, comment #1: 

Something I failed to mention --

If, after executing the series of commands noted in my original submission, you then perform a verify on d2/, you get a different error:

$ rdiff-backup --verify d2/
Warning: Cannot find SHA1 digest for file f3,
perhaps because this feature was added in v1.1.1
Every file verified successfully.


The patch I provided seems to fix this error as well.

Joe Steele <j_steele>
Fri 19 Jun 2009 09:30:56 PM UTC, original submission:  

The following series of commands will result in a "Hash for f3 missing, cannot check" error during the last restore command (using rdiff-backup 1.2.8):

#!/bin/bash
# Create a directory with 2 hardlinked files and back it up:
mkdir d1
touch d1/f1
link d1/f1 d1/f3
rdiff-backup d1 d2
# Now add a 3rd hardlinked file (but don't name it
# to be the first of the 3) and back up again:
link d1/f1 d1/f2
sleep 1
rdiff-backup d1 d2
# The following restore generates an error:
rdiff-backup -r now d2 d3


Furthermore, the metadata file for the second backup looks like this (note the discrepancy for 'NumHardLinks'):

File .
  Type dir
  ModTime 1245442871
  Uid 0
  Uname root
  Gid 0
  Gname root
  Permissions 493
File f1
  Type reg
  Size 0
  NumHardLinks 2
  Inode 79457
  DeviceLoc 64775
  SHA1Digest da39a3ee5e6b4b0d3255bfef95601890afd80709
  ModTime 1245442871
  Uid 0
  Uname root
  Gid 0
  Gname root
  Permissions 420
File f2
  Type reg
  Size 0
  NumHardLinks 3
  Inode 79457
  DeviceLoc 64775
  ModTime 1245442871
  Uid 0
  Uname root
  Gid 0
  Gname root
  Permissions 420
File f3
  Type reg
  Size 0
  NumHardLinks 2
  Inode 79457
  DeviceLoc 64775
  ModTime 1245442871
  Uid 0
  Uname root
  Gid 0
  Gname root
  Permissions 420

Furthermore, the restored files are linked differently from the original files:

# ls -li d[13]/
d1:
total 12
79457 -rw-r--r--  3 root root 0 Jun 19 16:21 f1
79457 -rw-r--r--  3 root root 0 Jun 19 16:21 f2
79457 -rw-r--r--  3 root root 0 Jun 19 16:21 f3

d3:
total 12
79492 -rw-r--r--  2 root root 0 Jun 19 16:21 f1
79492 -rw-r--r--  2 root root 0 Jun 19 16:21 f2
79493 -rw-r--r--  1 root root 0 Jun 19 16:21 f3
#


I've attached a file to this bug report which contains a possible patch for Hardlink.py that seems to fix this bug for me.

A real life example (on a CentOS-4 system) of this bug occurs if you have a few kernel-devel RPMs installed (for different kernels) and you back up the system, then you install another kernel-devel RPM for a newer kernel and you back up the system again, then you try to restore the most recent backup.  The restore will generate numerous (4,495 in my case) "Hash for foo missing, cannot check" errors.  The reason is because the kernel-devel RPM has a post-install script like this:

cd /usr/src/kernels/2.6.9-78.0.22.EL-i686
find . -type f | while read f; do hardlink -c /usr/src/kernels/*/$f $f ; done

As noted in the man page for hardlink:

"hardlink  traverses  one  or more directories searching for duplicate files.  When it finds duplicate files, it uses one of them as the master.  It then removes all other duplicates and places a hardlink for each one pointing to the master file.  This allows for conservation of disk space where multiple directories on a  single  filesystem contain many duplicate files."

So the hardlink command is adding new hardlinks to files that were already hardlinked, which triggers the bug in rdiff-backup.



Joe Steele <j_steele>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #21244:  Hardlink.py.revised-patch.txt added by j_steele (973B - text/plain - These are revised patches. Submitted 8/17/10.)
file #21245:  compare.py.patch.txt added by j_steele (3KiB - text/plain - These are revised patches. Submitted 8/17/10.)
file #18294:  Hardlink.py.patch.txt added by j_steele (473B - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ericzolf (Posted a comment)
  • -email is unavailable- added by drzraf
  • -email is unavailable- added by yenkwoon (Voted in favor of this item)
  • -email is unavailable- added by carloslp (Voted in favor of this item)
  • -email is unavailable- added by carloslp (Posted a comment)
  • -email is unavailable- added by j_steele (Submitted the item)
  •  

    There are 105 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-02-24 ericzolf StatusNone Duplicate
        Open/ClosedOpen Closed
    2015-12-04 drzraf Carbon-Copy- Added drzraf
    2010-08-17 j_steele Attached File- Added Hardlink.py.revised-patch.txt, #21244
        Attached File- Added compare.py.patch.txt, #21245
    2010-07-02 yenkwoon Carbon-Copy- Added yenkwoon
    2010-06-18 carloslp Carbon-Copy- Added carloslp
    2009-06-19 j_steele Attached File- Added Hardlink.py.patch.txt, #18294

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code