mainstoreBackup - Support: sr #108974, IsolatedMode with multiple series...

 
 

sr #108974: IsolatedMode with multiple series and bigger history

Submitter:  None
Submitted:  Sat 20 Feb 2016 01:01:08 AM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  2 - Minor Status:  None
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 21 Feb 2016 08:21:16 PM UTC, comment #3: 

Hi,

> wow, that's a quick response. And it's content makes it even better. Thanks a
> lot for the affirmation. Even though it will mean some work for me :-).


Just a few minutes before I read you post, I thought it's time to write some note
about that topic and to make some enhancements to the scripts. So I was well
prepared ... ;-)


> I know that replication would help, not only with ransomware. Off-site backups
> are a thing I still need to implement. Haven't done that yet because I don't have
>the hardware needed, and (more important) I don't have a place to put it other than
> my home, which would be pointless.


I have the same problem. But an offline backup helps in case of overvoltage because
of lightning (and that happened in about 20km distance from me).


> Regarding the error in the documentation:
> $ wget http://www.nongnu.org/storebackup/en/node33.html
> $ cp node33.html node33_corrected.html
> $ nano node33_corrected.html
> $ diff node33.html node33_corrected.html
> 78c78
> < storeBackupSetupIsolatedMode.pl -i isolateBackupDir -b backupDir
> ---
>
> >     storeBackupMergeIsolatedBackup.pl -i isolateBackupDir -b backupDir
> Or is this intentionally? The documentation doesn't mention an -i option for
>storeBackupSetupIsolatedMode.pl or storeBackupMergeIsolatedBackup.pl. But it
> would make more sense to me having storeBackupMergeIsolatedBackup.pl there.

Ah - ok, I missed what you mean. Will be corrected in the next release.


> Regarding the things you plan:
> 1. What good would be the --mvOnly option in storeBackupMergeIsolatedBackup.pl?
> Why would I use that instead of just mv? (I plan on using it with just -f and
> --force, since it has to run without interaction.)

storeBackupMergeIsolatedBackup.pl mostly is a luxury version of 'cp' (reading from
config file, doing several checks). It's simply the same for 'mv'.


> 2. Having a warning or any kind of feedback about the size of a backup would be
> definitely nice for some people, but I guess for another reason. It would create
> the possibility to make the most out of the available disk space. For checking whether
> a (new) backup was corrupted by ransomeware I would prefer storeBackupCheckBackup.pl.

I tought about getting information about an abnomal size of a backup because
many files changed. Maybe it's a good idea to have that kind of warning not only
for the size but for the number of files also!?
Naturelly, this can be used for whatever reason.
It's like always: Too much ideas an very limited time :-(


> 3. Ah, so I'm not the only one who dislikes writing documentation :-)

Se we're at least two :-)


> Additional Ideas?
> Having an option like '--useLastBackupsCount N' for storeBackupSetupIsolatedMode.pl,
> that does not only copy the very last backup to the isolated folder, but the last N
> backups would be nice. Not absolutely necessary though, since one could get around
> with calling the -e option several times in a for loop.

I think in reality, there's just one use case where this helps (for dedup): If you
restore a file from an old backup to the sourceDir (and do not change it then). But
this also means it's not from too far ago.

It may be more convenient to copy the meta data of other series you want to link to.
This works, you only need to call ...SetupIsoloated... in a (simple) loop with different
series.

Or do you have something different in mind?

Regards, Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Sat 20 Feb 2016 03:00:53 PM UTC, comment #2: 

Hi Heinz-Josef,

wow, that's a quick response. And it's content makes it even better. Thanks a lot for the affirmation. Even though it will mean some work for me :-).

I know that replication would help, not only with ransomware. Off-site backups are a thing I still need to implement. Haven't done that yet because I don't have the hardware needed, and (more important) I don't have a place to put it other than my home, which would be pointless.

Regarding the error in the documentation:
$ wget http://www.nongnu.org/storebackup/en/node33.html
$ cp node33.html node33_corrected.html
$ nano node33_corrected.html
$ diff node33.html node33_corrected.html
78c78
<     storeBackupSetupIsolatedMode.pl -i isolateBackupDir -b backupDir
---

>     storeBackupMergeIsolatedBackup.pl -i isolateBackupDir -b backupDir


Or is this intentionally? The documentation doesn't mention an -i option for storeBackupSetupIsolatedMode.pl or storeBackupMergeIsolatedBackup.pl. But it would make more sense to me having storeBackupMergeIsolatedBackup.pl there.

Regarding the things you plan:
1. What good would be the --mvOnly option in storeBackupMergeIsolatedBackup.pl? Why would I use that instead of just mv? (I plan on using it with just -f and --force, since it has to run without interaction.)
2. Having a warning or any kind of feedback about the size of a backup would be definitely nice for some people, but I guess for another reason. It would create the possibility to make the most out of the available disk space. For checking whether a (new) backup was corrupted by ransomeware I would prefer storeBackupCheckBackup.pl. Or did you mention this with another intension in mind?
3. Ah, so I'm not the only one who dislikes writing documentation :-)

Additional Ideas?
Having an option like '--useLastBackupsCount N' for storeBackupSetupIsolatedMode.pl, that does not only copy the very last backup to the isolated folder, but the last N backups would be nice. Not absolutely necessary though, since one could get around with calling the -e option several times in a for loop.

Greetings

Lookbehind

Anonymous
Sat 20 Feb 2016 07:57:00 AM UTC, comment #1: 

Hi,

it's absolutely perfect what you've written. I told the same to several people as a protection against those crypto trojans which may be a problem to Linux / Unix environments in the future also - or which are a problem regarding storeBackup for those using cygwin to run storeBackup from windows.

Just a few remarks and answers to you questions:

> The flaw:
> ... So if, for whatever reason, one of them goes haywire, it could delete
> ore overwrite ALL the backups. Even though this is pretty unlikely to
> happen, I would like to do something about that.


Just to mention: another protection against this is replication. But that's another story.

> Experiments:
> ... The documentation only mentions that this can be done with the
> very last backup, not the last 5, and not with ones of other series.)


Seems the documentation writer was somewhat lazy :-) . You can copy as many backup meta data from whatever series you want for isolated backup. It's only important that the directory structure is exactly the same as in you "real" backupDir.
The important point only is: Never delete a specific backup in backupDir which is referenced in isolated mode.

The reason why this all works is very simple: Apart from setting the hard links (which can be "out sourced" to storeBackupUpdateBackup.pl), only meta data is used to run a backup. And exactly that meta data is copied by storeBackupSetupIsolatedMode.pl. That's all. And this means, that you also can simple run the command "mv" instead of storeBackupMergeIsolatedBackup.pl if you isolated backup is on the same filesystem as backupDir. storeBackupMergeIsolatedBackup.pl just copies the files and does some additional checks so the data is not copied to a wrong place (if I remember well).

Additionally, if you want to "protect" against those crypto trojans (or against big manipulations in you sourceDir), you can check the size of the backup if some limit is reached. You may use that line from the log, e.g.:
STATISTIC 2016.02.20 02:01:55 25836 sum of target new = 12k (12499)


> By the way, there is an error in the documentation. In chapter "6.8
> storeBackupMergeIsolatedBackup.pl" for the part without a config file,
> there is "storeBackupSetupIsolatedMode.pl" written, where probably
> "storeBackupMergeIsolatedBackup.pl" is meant. It's in both the German
> (page 40) and the English (page 37) documentation.


From my understanding it's fine. Maybe I missed something!? Can you send me a corrected version of the part you mentioned (maybe via email)?


What I should do:
Just several minutes before I read you email I thought I should do the following:
1. Add an option like "--mvOnly" to storeBackupMergeIsolatedBackup.pl
2. Add an option like "--limitSize" to storeBackup.pl which writes a WARNING or and ERROR is some limits are hit by a backup. By this way you get the information if too much new data was saved.
3. Worst of all: Add documentation :-( .

Maybe you have some additional ideas?



Heinz-Josef Claes <hjclaes>
Group administrator
Sat 20 Feb 2016 01:01:08 AM UTC, original submission:  

Hey there,

in the process of making my personal backup solution a bit more secure, I wanted to try out abusing isolated mode for my needs. Despite my concerns, that I got from reading the documentation, it seams to work surprisingly well. At least in my test setup. But before using this in a productive environment, I wanted to ask if there is anything I might have missed.

Situation as is:
I have a little server in my basement and several computers (pcs, laptops, all Linux) that are sending their backups there using StoreBackup 3.5 with lateLinks, lateCompress and doNotDelete. I'm using an NFS share on the server that only gets mounted when needed. All the machines link their backups between each other considering the last 5 backups of their own series and the last two of all the other series. Every night the server does run storeBackupUpdateBackup.pl and storeBackupDel.pl to do the actual linking and cleaning out old backups. To restore from a Backup, I use an SMB share with read only access to the backups.
Nice setup, worked for more than 2 years that way without a hiccup.

The flaw:
Even though it works like a charm, there is one potential problem. All machines, while doing their backup, have write access to all the other backups, even to the ones of other machines. So if, for whatever reason, one of them goes haywire, it could delete ore overwrite ALL the backups. Even though this is pretty unlikely to happen, I would like to do something about that.

The plan:
Make the actual backups available read only on the network (SMB) and use the isolated mode feature to write new backups to a separate NFS share. At night, merge those isolated backups into the big pool and run storeBackupUpdateBackup.pl.
This way a potential evil machine could only destroy the backups of one day (which is sad, but I could live with that) and doesn't have write access to all the other ones.

Experiments:
So I set up a little test environment for 3 backup series with some fake data (./sourceA, B and C), a destination for the real backups (./realBackups) and one for the isolated backups (./isolatedBackups). Then generated config files with lateLinks and lateCompress as well as otherBackupSeries (0-5:seriesA 0-2:seriesB 0-2:seriesC). With that setup I created some backups directly to ./realBackups and ran storeBackupUpdateBackup.pl, just to have something to work with.
Then I ran storeBackupSetupIsolatedMode.pl -f seriesA.conf -t ./isolatedBackups to get a isolated config file. I took the lines for backupDir and mergeBackupDir and merged them into a copy of the original seriesA.conf (made it my_isol_seriesA.conf). Then made copys of this for seriesB and C and changed the values for sourceDir, series and otherBackupSeries accordingly.
(Note: otherBackupSeries for seriesA is "0-5:seriesA 0-2:seriesB 0-2:seriesC" again, even though this is the my_isol_seriesA.conf. The documentation only mentions that this can be done with the very last backup, not the last 5, and not with ones of other series.)
To have those older backups available I used storeBackupSetupIsolatedMode.pl -f my_isol_seriesA.conf -e 2016.02.19_21.20.05 with multiple backups and for all series. Afterwards I did multiple backups of all series in isolated mode with changing datasets and even coping data from sourceA to B and C. Then I ran storeBackupMergeIsolatedBackup.pl -f my_isol_seriesA.conf for all series and storeBackupUpdateBackup.pl right afterwards to integrate these Backups into ./realBackups.
And. It. Just. Worked! Looking at the inodes, they even got linked through the different series.

Finally the question:
Did I miss something? Is there anything that probably will go wrong with a setup like this? Or will my next step be writing some scripts to automate this?

By the way, there is an error in the documentation. In chapter "6.8 storeBackupMergeIsolatedBackup.pl" for the part without a config file, there is "storeBackupSetupIsolatedMode.pl" written, where probably "storeBackupMergeIsolatedBackup.pl" is meant. It's in both the German (page 40) and the English (page 37) documentation.

At last:
Thank you!
Thank you for taking the time to read this, much to long, support question.
And thank you for this amazing backup tool.

Greetings

Lookbehind

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code