mainstoreBackup - Support: sr #109248, Lock-File ignored? / deleted to...

 
 

sr #109248: Lock-File ignored? / deleted to early?

Submitter:  None
Submitted:  Sun 05 Feb 2017 01:58:00 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 07 Feb 2017 06:11:19 PM UTC, comment #2: 

Hi,

no, the Lock-File is a local file. In fact it's the default /tmp/storeBackup.lock, that I explicitly set via --lockFile. No need to have that on the target storage-space.

I avoided the problem by having my wrapper-srcipt look for PIDs attached to storeBackup.pl before launching the actual backup, and abort, when it finds one.

I don't know whether I would be able to test your fix. Because I am still unable to cause the issue on purpose. And waiting for it to happen naturally can take a month, or two, or three, ... It's not like this happens every time. But I noticed it several times now and thought I might as well tell you about it.

Greetings

Lookbehind

Anonymous
Mon 06 Feb 2017 09:51:19 AM UTC, comment #1: 

Hi,

do you read or write the lock file via NFS? In this case, maybe buffering is a problem!?

Nevertheless, I changed the order of execution to:

write backup -> sync -> write 'finished' -> write linkToRecent
          -> delete old backups -> start postcommand

I also corrected a bug in that area (see Changelog)

Don't know if this helps or matters. If you send me an email, I'll send you the new version (which I'm using since several months).

Regards,
Heinz-Josef

Heinz-Josef Claes <hjclaes>
Group administrator
Sun 05 Feb 2017 01:58:00 PM UTC, original submission:  

Hi there,

I've come across a strange problem, that I am unable to reproduce on purpose to look further into it. But had it happen the last few month every once in a while.

Base: A pretty slow machine: Pentium M 1,74 GHz, 512MB RAM, but 4 Harddrives, making two RAID 1 volumes with mdadm and connected via LVM. All on Debian Wheezy. StoreBackup-Version is 3.5 and runs over Network with lateLinks and lateCompress.

The machine starts a check of the RAID Systems every first Sunday of a month. This takes nearly the entire day, because of the unbelievable speeds of the machine. Which normally is no problem, because the machine it self stays operable. The Backup takes a bit longer though. About 2 hours. Which also is not a big Problem. On Sunday 5 am, there is nobody really needing the machine. So let it take its time.

The Problem occurs when the backup is nearly done. Log-File stated the end 12 seconds ago, "backup.Finished" is set in .storeBackupLinks/ but storeBackup.pl is still running. Starting an new backup in this situation can cause the new backup to realy start, but the old backup-process never to end. Sadly I have no way of knowing if the "old" lockfile was still in place at this moment. But --lockFile points to the same file for both processes, now containing the PID of the new backup-process.
storeBackup.pl is run with nice -n 10, I should add.

Ok, so I am going to have an extra Backup, that I didn't ask for. What of it? Where is the Problem?
Well, this keeps adding on. The first backup process, even though the log file states it's end, never stops running. Near the end of the second backup-process the next one starts, and the second one also will never exit, then the third, and when the fourth backup kicks in, this incredible powerfull machine starts to even drop tcp packets, because its CPU doesn't know what to do first anymore. (Needless to say, that RAM and SWAP are everything but empty)
Locally (you can forget about ssh at this moment) logging in as root and killing the processes by hand helps, but takes about 20 minutes because of the incredible response times.

Yes, my cron-job needs improvement. I'm working on it, and it will most certainly avoid the problem.
Never the less, reading the documentation I understand it that way, that --lockFile is supposed to stop storeBackup from starting up, when another process of storeBackup is still running. This somehow seams not to work in all cases. You may want to take a look into this.

As I said, sadly I was not able to reproduce this on purpose, and it's a pretty rare problem. So it might not be the most easiest of tasks to figure out, what exactly goes wrong. But still I wanted you to know about this.


Greetings

Lookbehind

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hjclaes (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code