bugrdiff-backup - Bugs: bug #27296, RFE: hardlink sharing between...

 
 

bug #27296: RFE: hardlink sharing between multiple archive backups

Submitter:  Joshua Jensen <joshmule>
Submitted:  Fri 21 Aug 2009 04:05:32 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 12 Feb 2016 10:02:07 AM UTC, comment #2: 

Closing this as a de-duping filesystem is a better idea. ZFS is even more mature now.
Please repoen at https://github.com/sol1/rdiff-backup if you want to work on it more.

david kempe <davekempe>
Group administrator
Thu 19 Aug 2010 07:09:29 PM UTC, comment #1: 

You should investigate BTRFS - it has COW object cloning that should be able to de-duplicate those files.

If rdiff-backup does it, it needs to remember to un-link the file when (before) it changes, then relink other versions for further backups.

But with COW in BTRFS this is easier. I'm not sure exactly how to set it up though, since I've never used it, just read about it.

Ariel <ars>
Fri 21 Aug 2009 04:05:32 PM UTC, original submission:  

I ask because I have 2000 (yes, two thousand) machines to back up.  At least 90% of data on each machine is the same... basic Linux filesystems, mostly / and /var and /usr partitions.

I wonder if it would be worthwhile to add a feature to rdiff-backup that would look to hardlink files that different rdiff-backup archives have in common?

Right now, the hard link savings is only per machine's, say, /usr
partition.  Sure, the incremental backup from the same machine
benefits from this; only any differences are recorded.

But right next to that archive directory on the backup server I have a thousand other archvies with a /usr archive that is mostly identical.  Why not have the option to make rdiff-backup
be able to use some sort of higher level "all these archives can share hardlinks" capability?  The space saving would be staggering for multihost backups.

I was thinking that rdiff-backup could just keep an additional
hardlink-only list of files/checksums that any individual run could reference, thus giving it a perspective outside of that individual backup.  When it found a file that was already on that global/group list, it need not transfer it, only hardlink to it.  This could make storing similar files MUCH more space and network efficient.

Thoughts?

Joshua Jensen <joshmule>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by davekempe (Posted a comment)
  • -email is unavailable- added by ars (Posted a comment)
  • -email is unavailable- added by joshmule (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-02-12 davekempe Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code