mainstoreBackup - Support: sr #108591, Simple Replication

 
 

sr #108591: Simple Replication

Submitter:  Jonas <sanoj>
Submitted:  Tue 03 Jun 2014 06:40:59 AM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  2 - Minor Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 13 Jun 2014 02:53:02 AM UTC, comment #5: 

I have to correct myself - the friend of mine copied 80 million items (not 10 million) :-) in about two days.

Time for copying depends a lot on bandwidth, latency of the disks and maybe if and where you put your /tmp (if you use --saveRAM, which is faster if combined with a ram disk).

The only chance to interrupt the copying process at the moment is to use the unix tool 'kill':
 kill -SIGSTOP <pid>
and
 kill -SIGCONT <pid>
should do the job (where <pid> is the process ID of linkToDirs.pl)

Naturally, you cannot unmount the disks or reboot during the kill commands. And you should not change the data at that time frame.

Hope your copy is finished successfully meanwhile.
Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Wed 11 Jun 2014 07:56:14 AM UTC, comment #4: 

ah ok, thanks!

I try to copy about 5 mio... but it takes days!
A nice feature would be to interrupt and resume the process...

Thanks for your help!
Jonas

Jonas <sanoj>
Sun 08 Jun 2014 11:00:41 AM UTC, comment #3: 


> Can I merge them if I just copy all the backup content from HDD2
> to HDD1 using linkToDirs? Does storebackup also recognize all
> the right dates then to automatically keep appropriate backups?


Yes, no problem. To make thinks easy and clear, I think best is to make it in the following way:

In the example I assume mount points are /disk1 for HDD1 and /disk2 for HDD2.
Backup is always in subdirectory 'stbu', so in /disk1/stbu and /disk2/stbu.

1. make a sub directory on HDD1, eg /disk1/HDD2
2. copy / hard link the files:
linkToDirs.pl /disk2/stbu -w /disk1/stbu -t /disk1/HDD2 -D -P 5000,1m
(Please double check the line!)
This should copy / hardlink /disk2 to /disk1/HDD2.
If something went wrong, you can simply delete the contents of /disk1/HDD2 and restart. If everything's fine, you can move (mv) the backup directories in/disk1/HDD2 to the proper place in /disk1/stbu.

> And another question: If I copied the backups using cp can I
> replace identical copies by hardlinks afterwards (something
> like find all identical files in this folder (recursively) and
> replace them by hardlinks)


No. There maybe tools to do this, but storeBackup doesn't support this feature (at least at the moment). The reason why I made it this way (linkToDirs.pl) are:
- You need less space on the target directory. (To be precise: you do not need temporarily a lot of space.)
- If you change "in-place", interrupting the process may result in a change (or loss) of at least one file.



How much items do you copy? A few days ago a friend of mine copied about 10 million items in that way.

Regards, Heinz-Josef



Heinz-Josef Claes <hjclaes>
Group administrator
Sun 08 Jun 2014 10:36:03 AM UTC, comment #2: 

Hi Heinz-Josef,

yes, that helps, thanks!
I have a little extension of the questions: If I have two different storage devices with storeBackup-backups of the same series of different timepoints

for example

HDD1 - series x: backup at time A, backup at time C
HDD2 - series x: backup at time B, backup at time D, backup at time E

Can I merge them if I just copy all the backup content from HDD2 to HDD1 using linkToDirs? Does storebackup also recognize all the right dates then to automatically keep appropriate backups?

And another question: If I copied the backups using cp can I replace identical copies by hardlinks afterwards (something like find all identical files in this folder (recursively) and replace them by hardlinks)

Thanks for your help and your great tool!
Jonas

Jonas <sanoj>
Tue 03 Jun 2014 07:05:56 PM UTC, comment #1: 

Hi Jonas,

> If I replicate a backup from one media to another using
> linkToDirs, I get a exact copy of the backup including all
> links, all checksums, all time-constraints, ..., right?

I think 'replication' is not the best of all possible words for this. It's more a kind of copying with additional hard linking identical of identical files also.
But to answer your question, You'll get an exact copy of the source. But maybe it's smaller than the original because it may find different files with the same contents which are hard linked in the target (copied) directory.

> Can I use the replicated backup as new masterbackup and add
> newer incremental backups without any further adjustments of the
> config files?

Long answer: Storebackup's functionality doesn't care if files in the backup are hard linked or not (hard linking is just for saving memory, but has nothing to do with the functionality). So, from the point of view of storeBackup, the copy behaves like the original.
Short answer: YES. :-)

Hope this answers your question.

Regards, Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Tue 03 Jun 2014 06:40:59 AM UTC, original submission:  

Hi,

I have a question just to be sure: If I replicate a backup from one media to another using linkToDirs, I get a exact copy of the backup including all links, all checksums, all time-constraints, ..., right?

Can I use the replicated backup as new masterbackup and add newer incremental backups without any further adjustments of the config files?

Thanks for the help!
Bye

Jonas <sanoj>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)
  • -email is unavailable- added by sanoj (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code