mainstoreBackup - Support: sr #108338, Combine two master backups into one

 
 

sr #108338: Combine two master backups into one

Submitter:  None
Submitted:  Thu 11 Jul 2013 02:35:10 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 12 Jul 2013 07:04:43 PM UTC, comment #8: 

Heinz-Josef, your advice was spot on!

So this was an rtfm after all, I guess I just wasn't imagining that storeBackup would really be THAT powerful so I did not search for this option thoroughly enough.
Sorry for that, and many thanks again for the quick help.

Regards
Robert

 Anonymous
Fri 12 Jul 2013 03:22:30 PM UTC, comment #7: 

Hello Heinz-Josef

thanks again for the quick reply. I am not sure I understood you, but the 'otherBackupSeries' option is probably what I am looking for.

I will play around with it a bit and let you know.

Thanks again
Robert

 Anonymous
Fri 12 Jul 2013 01:07:55 PM UTC, comment #6: 

To be honest, I didn't get it completely, but maybe I can help.

storeBackup is working in the following way:


There is one backupDir, eg. /srv/backup

Below the backupDir there are the series. A series is nothing else than a directory below the backupDir - or maybe (undocumented feature!) multiple sub directories, eg. systemconfig/server. In the last example the full path would be /srv/backup/systemconfig/server. Below that path storeBackup would create the "timestamp directories" with the individual backups.


I now assume, that you want to change your layout of the backups completely and relink the files. Everything is on one partition.

1. create a new backupDir

2. mv all existing series into the new backupDir (It does not really matter how you organize it.) BUT it's important you only have series directories and their content there. Otherwise, storeBackup will search recursively through the rest for timestamp directories!

3. create the new series structure below backupDir with mkdir

4. start the backups for the new series layout without any settings of "otherBackupSeries". StoreBackup.pl will search all series and build the index with the files from the last backup in each series.

5. After the first backup for each new series, change otherBackupSeries so the storeBackup.pl will only create an index for the files of that series. Example for series systemconfig/server:
otherBackupDirs = 0:systemconfig/server


Hope this helps.
Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Thu 11 Jul 2013 09:47:55 PM UTC, comment #5: 

The format of my directory layouts got messed up, so I am posting them again... Hope this is better.

Current layout

/srv/backup/
---------> mylaptop
----------------->snapshot
-------------------------> home
-------------------------> systemconfig
----------------->archive
------------------------default
---------> mywifeslaptop
----------------->snapshot
-------------------------> home
-------------------------> systemconfig
----------------->archive
------------------------default
---------> server
----------------->snapshot
-------------------------> systemconfig
----------------->archive
------------------------default

Desired layout

/srv/backup/
---------> userdata
-----------------> me
-----------------> mywife
---------> systemconfig
-----------------> mylaptop-distro1
-----------------> mylaptop-distro2
-----------------> mylaptop-distro3
-----------------> mynetbook-distro4
-----------------> mywifeslaptop
-----------------> server
/srv/archive (on different disk)
---------> default

 Anonymous
Thu 11 Jul 2013 09:44:27 PM UTC, comment #4: 

Hello Heinz-Josef

first of all, thank you for the quick and elaborate answer! (And for giving this great tool away for free in the first place)

So I have repeated all the steps you described with your example, and I think I understand what you are doing. But I also realize that I have asked my question imprecisely, and now you answered what I asked, which is not what I actually wanted to know :-)

I don't want to make that mistake again, so I will try to describe what I am trying to accomplish in more detail. Sorry if I am overdoing it. I'd understand if you did not have the time to read all of this :-)

The three individual backups don't have to be interlinked. They have no files in common. I guess I could easily move the three series together, without even needing linkToDirs.

But really I would like to combine them in a single series. Let me explain.

I have a server at home, which my laptop and my wife's laptop are syncing with every hour to backup the data through an rsync script. My current setup on the server has one backupdirectory for her laptop, one for mine, and one for the server's configuration files.

The exact layout is as follows:

/srv/backup/
-> mylaptop
->snapshot
-> home
-> systemconfig
->archive
default
-> mywifeslaptop
->snapshot
-> home
-> systemconfig
->archive
default
-> server
->snapshot
-> systemconfig
->archive
default

For each backupdirectory I created a series that was run by storebackup to archive the data. (Actually on the same disk. I am planning to move the archive to a second disk, but first I have to sort out the restructuring I am currently working on)

For each of the three directories 'mylaptop', 'mywifeslaptop' and 'server' there is a storebackup profile that backs up 'snapshot' into 'archive', running every night.

Now I am using more than one distro on my laptop, and I am using other computers, so I want to organize the backup data differently.
Just one big backup dir, which is split into a subdir "userdata" and a subdir "systemconfig"
The userdata will contain my wife's data and my data. The systemconfig will have one subfolder for each distro that is installed in our household, so like 3 for my laptop, another one for a netbook, and one more for her laptop. In other words, its organized in a completely different way.
That one big backup dir should be archived by storebackup onto the another disk (which is already sitting in another slot of the server, waiting to start being used).

/srv/backup/
-> userdata
-> me
-> mywife
-> systemconfig
-> mylaptop-distro1
-> mylaptop-distro2
-> mylaptop-distro3
-> mynetbook-distro4
-> mywifeslaptop
-> server
/srv/archive (on different disk)
-> default

The nightly storebackup shall now backup /srv/backup/ into /srv/archive.

So after going through your example my thinking goes like this:
I could move the three series into /srv/archive and rename them in the process:

mv /srv/backup/mylaptop/archive/default /srv/archive/series_mylaptop
mv /srv/backup/mywifeslaptop/archive/default /srv/archive/series_mywifeslaptop
mv /srv/backup/server/archive/default /srv/archive/series_server

Then I would create a new directory
mkdir /srv/archive/series_combined

That should be the new target from now on, but when running the combined series for the first time it would just copy the whole directory /srv/backup, rather then hard linking to all the files that already exist in the other 3 series...
Is there any way to uses linkToDirs to set up this new combined_series?

Many thanks again,
Robert

 Anonymous
Thu 11 Jul 2013 07:54:58 PM UTC, comment #3: 

I hope the source is also on another disk ;-) .

Ok, I think it's not possible with the existing tools in a perfect way - which means without temporarily creating new files. So the question is if you have enough space on that disk. I make an example to make it clear how to link the backups with linkToDir.pl.

Before doing anything similar to the steps described above, you have to run storeBackupUpdateBackup.pl if you used option lateLinks. If you didn't use that option, there's no need to do so. I also assume, you are not using replication!


The following commands just create three different backup series in different backup trees without any linking between them. The source directory is "s" and each backup contains the identical file "ls" plus one individual file called "backup". Backup2 and Backup3 also each contain a file called "backup23" which have the same contents:

mkdir /tmp/a
cd /tmp/a
mkdir s
mkdir -p backupDir1/series1 backupDir2/series2 backupDir3/series3
cp /bin/ls s
echo "backup1" > s/backup
storeBackup.pl -s s -b backupDir1 -S series1
echo "backup2" > s/backup
echo "backup23" > s/backup23
storeBackup.pl -s s -b backupDir2 -S series2
echo "backup3" > s/backup
storeBackup.pl -s s -b backupDir3 -S series3

In the following commands, I copy / link backup2 and backup3 to backup1. The target for this new copying / linking is directory backupDir1/NEW, just to make it clear that this is new:

mkdir backupDir1/NEW
linkToDirs.pl -w backupDir1/series1 -t backupDir1/NEW backupDir2/series2 backupDir3/series3 --saveRAM

(You might need option saveRAM or not, but it's most save to use it.)

The linkToDirs.pl command above has the following output:
BEGIN     2013.07.11 21:29:05 16737 copy/link <backupDir2/series2><backupDir3/series3> to <backupDir1/series1> to <backupDir1/NEW>
VERSION   2013.07.11 21:29:05 16737 linkToDirs.pl, 3.4
INFO      2013.07.11 21:29:05 16737 setting atime, mtime of directories ...
STATISTIC 2013.07.11 21:29:05 16737 read 4 items; created 6 dirs, 3 hard links, 7 copied; calced 14 md5 sums,
END       2013.07.11 21:29:05 16737 copy/link <backupDir2/series2><backupDir3/series3> to <backupDir1/series1> to <backupDir1/NEW>

(You might want to use option -P with your real backup so you see something's going on.)

As you can see, some files are copied. That means, in worst case you need additionally the space for backup2 and backup3 temporarily.

Have a look at series2 and series3 in directory NEW. All identical files in backup1/series1, backup1/NEW/series2 and backup1/NEW/series3 are hard linked now.

So let's correct the location of the directories:

cd /tmp/a/backupDir1
mv new/* .
rmdir NEW

All series (1,2,3) are now below backup1

We also can change the name of backup1 to something more generic:

cd ..
mv backupDir1 backupDir

Run storeBackupCheckBackup.pl on backupDir:

storeBackupCheckBackup.pl -c backupDir
BEGIN     2013.07.11 21:47:47 17317 checking backups in </tmp/a/backupDir>
VERSION   2013.07.11 21:47:47 17317 storeBackupCheckBackup.pl, 3.4
INFO      2013.07.11 21:47:47 17317 scanning directory <backupDir> for existing backups
INFO      2013.07.11 21:47:47 17317 scanning directory <backupDir/series3> for existing backups
INFO      2013.07.11 21:47:47 17317 scanning directory <backupDir/series2> for existing backups
INFO      2013.07.11 21:47:47 17317 scanning directory <backupDir/series1> for existing backups
STATISTIC 2013.07.11 21:47:47 17317 found 3 backup series, 3 backups, 0 renamed backups
INFO      2013.07.11 21:47:47 17317 consistency check finished successfully
INFO      2013.07.11 21:47:47 17317 found no references to backups from lateLinks that need storeBackupUpdateBackup run
INFO      2013.07.11 21:47:47 17317 backup directories to check
INFO      2013.07.11 21:47:47 17317   /tmp/a/backupDir/series1/2013.07.11_21.27.59
INFO      2013.07.11 21:47:47 17317   /tmp/a/backupDir/series2/2013.07.11_21.28.01
INFO      2013.07.11 21:47:47 17317   /tmp/a/backupDir/series3/2013.07.11_21.28.02
INFO      2013.07.11 21:47:47 17317 -- checking </tmp/a/backupDir/series1/2013.07.11_21.27.59> ...
INFO      2013.07.11 21:47:47 17317 -- checking </tmp/a/backupDir/series2/2013.07.11_21.28.01> ...
INFO      2013.07.11 21:47:47 17317 -- checking </tmp/a/backupDir/series3/2013.07.11_21.28.02> ...
INFO      2013.07.11 21:47:47 17317 -- no WARNINGS OCCURRED DURING THE CHECK! --
INFO      2013.07.11 21:47:47 17317 -- no ERRORS OCCURRED DURING THE CHECK! --
END       2013.07.11 21:47:47 17317 checking backups in </tmp/a/backupDir>

If everything is fine you can delete backupDir2 and backupDir3

Then, you have to adjust your scripts or configuration file to the new backupDir (and maybe series if you also want to change their names).

Hope this helps.

Heinz-Josef

 Heinz-Josef Claes <hjclaes>
Group administrator
Thu 11 Jul 2013 07:04:58 PM UTC, comment #2: 

Hi

yes, they are. The directory structure is as follows:

/some/path/on/one/partition/subdir1/subsubdir1/default
/some/path/on/one/partition/subdir2/subsubdir2/default
/some/path/on/one/partition/subdir3/subsubdir3/default

The source directory for the new combined backup is on another partition of the same machine.

Regards

 Anonymous
Thu 11 Jul 2013 04:34:51 PM UTC, comment #1: 

Hi,

yes, it's at least similar.

Are your different backupDir's on the same partition?

Regards, Heinz-Josef

 Heinz-Josef Claes <hjclaes>
Group administrator
Thu 11 Jul 2013 02:35:10 PM UTC, original submission:  

Hi

I am not sure if this might be a similar request as item 107107.

The question is can I combine two (or more) individual master backups into 1? I saw the Isolated / offline mode, but it seems this requires to plan the merge upfront when setting up the backup profile.
I have 3 backups that were set up independently. Now I have restructured the data organization in away that the backup process could be reduced to one source and one target, just running from a single profile.
I could obviously wipe the backup and restart from zero, but I would like to keep my history without having to double the size of the backup.
Is there any way to do this?

Thank you in advance!

 Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-09-30 hjclaes StatusNone Done

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-f13d.
    Corresponding source code