Hi, let me just add:

You wrote about lots of files and long backup duration (or long updateBackup duration). Maybe some of these hints help:

- as far as I know, ext4 is by far the fastest file system for making hard links (don't know about btrfs, but that's not the right one for making backups up to now I think)

- make an entry in /etc/fstab for /tmp as a ramdisk, eg:
none            /tmp    tmpfs           size=2048M
and use option saveRAM (with storeBackup.pl). The implementation of hashes in perl is pretty poor: needs lot of memory and is slow. If you use saveRAM then Berkeley DB files are used in the ram disk (--tmpdir). The result is that you need less memory and that it's faster. Because you need less memory, more memory is available for caching which should speed up thinks additionally.

- Here:
http://code.google.com/p/compressible/wiki/StoreBackupTips
you find some hints about "Memory usage and OS tuning".
I never tested this, but maybe it helps.

Hope this helps!?
Heinz-Josef

StoreBackupUpdateBackup comes with a warning that it should not be run in parallel with StoreBackup.  To that end I use lockfiles and run at different times.  Mostly this is OK because of the timing separation.

However, on occasion, the backup takes longer than usual.  In these instances, the update part starts running and happily deletes the lockfile from the remote process.  (In fact, the reason it's taking so long this time is because of this locking behavior messing up my backups until the remote storage ran out of space, and now there's a ton of old ones to delete at once that never got Update ran... basically it spiralled out of control.)

I have made both servers respect the lockfile, even if they cannot find a local process, by inserting this into the 'else' case for checkLockFile:
$prLog->print('-kind' => 'E',
  '-str' => ["cannot start, old (REMOTE) instance with pid " .
     "<$pid> may be running"],
  '-exit' => 1);

I don't know if this is a bug persay, but it was unexpected behavior to me.  I expected that if a lockfile existed, then storeBackup should not proceed.

I know that in power-loss or similar situations, this may cause a problem, but I'd rather have it do the right thing when everything is fine, and only mess up in extraordinary situations.