bugrdiff-backup - Bugs: bug #38325, --check-destination-dir memory...

 
 

bug #38325: --check-destination-dir memory required is not constant

Submitted by:  Francois BERENGER <HappyCrow>
Submitted on:  Thu 14 Feb 2013 05:20:55 AM UTC  
Votes:  10  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Postponed
Privacy: PublicAssigned to: None
Open/Closed: Open

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 30 Aug 2013 10:22:41 AM UTC, comment #1:

Hi,

the command:
rdiff-backup --check-destination-dir /backup/hostname
fails after consuming over 4Gb of real memory (it used zero swap memory). There are about 13 million files (709 Gb) of data.

Here the traceback (verson 1.2.8 on Debian Squeeze):

Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 30, in <module>
rdiff_backup.Main.error_check_Main(sys.argv[1:])
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 282, in take_action
elif action == "check-destination-dir": CheckDest(rps[0])
File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 872, in CheckDest
dest_rp.conn.regress.Regress(dest_rp)
File "/usr/lib/pymodules/python2.6/rdiff_backup/regress.py", line 71, in Regress
for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
File "/usr/lib/pymodules/python2.6/rdiff_backup/regress.py", line 197, in iterate_meta_rfs
for raw_rf, metadata_rorp in collated:
File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line 100, in Collate2Iters
try: relem2 = riter2.next()
File "/usr/lib/pymodules/python2.6/rdiff_backup/eas_acls.py", line 207, in join_ea_iter
for rorp, ea in rorpiter.CollateIterators(rorp_iter, ea_iter):
File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line 103, in Collate2Iters
for relem1 in riter1:
File "/usr/lib/pymodules/python2.6/rdiff_backup/eas_acls.py", line 566, in join_acl_iter
for rorp, acl in rorpiter.CollateIterators(rorp_iter, acl_iter):
File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line 103, in Collate2Iters
for relem1 in riter1:
File "/usr/lib/pymodules/python2.6/rdiff_backup/metadata.py", line 274, in iterate
for record in self.iterate_records():
File "/usr/lib/pymodules/python2.6/rdiff_backup/metadata.py", line 283, in iterate_records
next_pos = self.get_next_pos()
File "/usr/lib/pymodules/python2.6/rdiff_backup/metadata.py", line 266, in get_next_pos
newbuf = self.fileobj.read(self.blocksize)
File "/usr/lib/python2.6/gzip.py", line 219, in read
self._read(readsize)
File "/usr/lib/python2.6/gzip.py", line 272, in _read
self._add_read_data( uncompress )
File "/usr/lib/python2.6/gzip.py", line 289, in _add_read_data
self.extrabuf = self.extrabuf + data
MemoryError

Niccolo Rigacci <niccolo>
Thu 14 Feb 2013 05:20:55 AM UTC, original submission:

Hello,

I think this option should execute using a constant
amount of memory.

I have a user directory which cannot be backup anymore
neither checked because the memory it requires is huge,
so at some time Linux kills the process.

Thanks,
Francois.

Francois BERENGER <HappyCrow>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by davekempe (Updated the item)
  • -unavailable- added by niccolo (Posted a comment)
  • -unavailable- added by niccolo (Voted in favor of this item)
  • -unavailable- added by HappyCrow (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 10 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 12 Feb 2016 09:43:39 PM UTCdavekempeStatusNone=>Postponed
    Fri 30 Aug 2013 08:48:01 AM UTCniccoloCarbon-Copy-=>Added niccolo

    Back to the top


    Powered by Savane 3.1-cleanup1