bugrdiff-backup - Bugs: bug #38325, --check-destination-dir memory...

 
 

bug #38325: --check-destination-dir memory required is not constant

Submitter:  Francois BERENGER <HappyCrow>
Submitted:  Thu 14 Feb 2013 05:20:55 AM UTC
Votes: 10
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Postponed
Privacy:  Public Assigned to:  None
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 20 May 2022 05:44:28 PM UTC, comment #2: 

This issue has been closed because it hasn&#039;t been touched for years, as the development continued on GitHub. If the issue is still present in a recent version, please re-open it under https://github.com/rdiff-backup/rdiff-backup/issues after having made that it&#039;s not already reported there.

Eric L. <ericzolf>
Group administrator
Fri 30 Aug 2013 10:22:41 AM UTC, comment #1: 

Hi,

the command:
rdiff-backup --check-destination-dir /backup/hostname
fails after consuming over 4Gb of real memory (it used zero swap memory). There are about 13 million files (709 Gb) of data.

Here the traceback (verson 1.2.8 on Debian Squeeze):

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 282, in take_action
    elif action == "check-destination-dir": CheckDest(rps[0])
  File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 872, in CheckDest
    dest_rp.conn.regress.Regress(dest_rp)
  File "/usr/lib/pymodules/python2.6/rdiff_backup/regress.py", line 71, in Regress
    for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
  File "/usr/lib/pymodules/python2.6/rdiff_backup/regress.py", line 197, in iterate_meta_rfs
    for raw_rf, metadata_rorp in collated:
  File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line 100, in Collate2Iters
    try: relem2 = riter2.next()
  File "/usr/lib/pymodules/python2.6/rdiff_backup/eas_acls.py", line 207, in join_ea_iter
    for rorp, ea in rorpiter.CollateIterators(rorp_iter, ea_iter):
  File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line 103, in Collate2Iters
    for relem1 in riter1:
  File "/usr/lib/pymodules/python2.6/rdiff_backup/eas_acls.py", line 566, in join_acl_iter
    for rorp, acl in rorpiter.CollateIterators(rorp_iter, acl_iter):
  File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line 103, in Collate2Iters
    for relem1 in riter1:
  File "/usr/lib/pymodules/python2.6/rdiff_backup/metadata.py", line 274, in iterate
    for record in self.iterate_records():
  File "/usr/lib/pymodules/python2.6/rdiff_backup/metadata.py", line 283, in iterate_records
    next_pos = self.get_next_pos()
  File "/usr/lib/pymodules/python2.6/rdiff_backup/metadata.py", line 266, in get_next_pos
    newbuf = self.fileobj.read(self.blocksize)
  File "/usr/lib/python2.6/gzip.py", line 219, in read
    self._read(readsize)
  File "/usr/lib/python2.6/gzip.py", line 272, in _read
    self._add_read_data( uncompress )
  File "/usr/lib/python2.6/gzip.py", line 289, in _add_read_data
    self.extrabuf = self.extrabuf + data
MemoryError

Niccolo Rigacci <niccolo>
Thu 14 Feb 2013 05:20:55 AM UTC, original submission:  

Hello,

I think this option should execute using a constant
amount of memory.

I have a user directory which cannot be backup anymore
neither checked because the memory it requires is huge,
so at some time Linux kills the process.

Thanks,
Francois.

Francois BERENGER <HappyCrow>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ericzolf (Posted a comment)
  • -email is unavailable- added by davekempe (Updated the item)
  • -email is unavailable- added by niccolo (Posted a comment)
  • -email is unavailable- added by niccolo (Voted in favor of this item)
  • -email is unavailable- added by HappyCrow (Submitted the item)
  •  

    There are 10 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-05-20 ericzolf Open/ClosedOpen Closed
    2016-02-12 davekempe StatusNone Postponed
    2013-08-30 niccolo Carbon-Copy- Added niccolo

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code