mainstoreBackup - Support: sr #108315, hard links of empty files

 
 

sr #108315: hard links of empty files

Submitter:  None
Submitted:  Fri 07 Jun 2013 11:11:28 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  2 - Minor Status:  Done
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 30 Sep 2013 01:49:39 PM UTC, comment #4: 

added as faq (in next release)

Heinz-Josef Claes <hjclaes>
Group administrator
Sat 08 Jun 2013 11:00:54 AM UTC, comment #3: 

"My special problem (rsyncing from ext4 to btrfs) is a silly idea at least as far as the different max hardlink count is concerned."

Use linkToDirs.pl for this. Simply use
linkToDirs.pl <sourceDir> --targetDir <targetDir>

It will also link all identical files which have not been linked in your <sourceDir>.

Regards, Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Sat 08 Jun 2013 10:56:48 AM UTC, comment #2: 

Hi Heinz-Josef,

I have guilty conscience, spending so much of your time, but I hope that discussing it here has the side effect of building an FAQ.
(of course I absorbed storebackup.pdf documentation twice and searched in the service requests here)

Everything is answered. My special problem (rsyncing from ext4 to btrfs) is a silly idea at least as far as the different max hardlink count is concerned.

Thanks a lot!

lopiuh

Anonymous
Sat 08 Jun 2013 09:52:07 AM UTC, comment #1: 

For each file you need at least an inode, so hard linking zero byte files saves at least lots of inodes. This may have no result (if the filesystem (ext) has enough statically reserved or really saves some memory if the filesystem allocates them dynamically (reiserfs, btrfs) or you run out of inodes with ext filesystems.

Setting a new hard link should also be faster than creating a new inode.

For storeBackup, handling of hardlinks is no problem. It tries to create a hardlink, and if this is not successful, it creates a new file and hard links against this in the future. Running out of hardlinks simply means to create a new identical file. Because of this simple and stupid algorithm, the number of hard links a file system supports is nothing storeBackup has to care about. This behavior is different to typical Unix tools like cp, tar or rsync. If you copy a directory with lots of hardlinks to to a filesystem which does not support enough hardlinks, you will get errors (see also explanation to program linkToDirs.pl delivered with storeBackup which bypasses this limitation).

In case of btrfs. I would not use it for backup at the moment because I think it's not stable enough. But anyway, I made some test and its behavior seems to be very different from other filesystems regarding hardlinks. It has a very limited number of hardlinks in_one_directory . I ran out of hardlinks - all files in one directory - but was able to create additional hardlinks to those inode from another directory.
But anyway - because of storeBackups stupid algorithm, it can handle btrfs also efficiently.

Finally, I think there's no reason not-to-hard-link zero byte files.

Maybe I wrote too much stuff. Does this answer you question (at least the "save space" one) ;-) ?

Regards, Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Fri 07 Jun 2013 11:11:28 PM UTC, original submission:  

I remember playing with anti dupe tools (deduplication) that they normaly did not hardlink zero or "small" byte files.

Do we have problems with "hard link count" / too much hardlinks? I did a test on btrfs (rsyncing an [ext4] storebackup series) and it oopsed with  "too many hardlinks" i know ... I should not use rsync and would I have used storebackup to backup on a filesystem with lower max hardlinks storebackup would have started a new file and restart hardlinks.

Just an idea. Do we save space by hardlinking zero-byte files?

Thanks

lopiuh

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-09-30 hjclaes StatusNone Done

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code