bugConcurrent Versions System - Bugs: bug #16630, out of memory message during commit

 
 

bug #16630: out of memory message during commit

Submitter:  Frank Heitbrock <frankheitbrock>
Submitted:  Fri 19 May 2006 03:14:53 PM UTC
   
 
Category:  Bug Report Severity:  1 - Wish
Item Group:  None Status:  Confirmed
Privacy:  Public Assigned to:  None
Open/Closed:  Open Release: 
Fixed Release:  None Fixed Feature Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

  Spam posted by anonymous
Wed 14 Jun 2006 05:34:47 PM UTC, comment #3: 

Yes, that number is very close to the first one you reported (1,004,772,198 - 1,004,771,977 = 221 bytes, probably the number of bytes needed to preparse the archive header).  There is nothing that can be done on this end about this, short of rewriting the way CVS allocates and uses memory when parsing a file, or possibly adding binary diff code to the CVS server to avoid creating such large archives in the first place (upping your server memory or refraining from checking in such large binary files might be a solution you could implement without touching the CVS internals).  We would love to see a patch!

Derek Robert Price <dprice>
Group administrator
Wed 14 Jun 2006 11:55:37 AM UTC, comment #2: 

I have checked the size of the failing (,v) file in the archive today. The size is 1,004,772,198 bytes. I have renamed the failing file and checked it out again.
At the end of the new checkout, i got the follwoing error message from cvs:
"cvs:buffer.c:1384: stdio_buffer_shutdown:
Assertion 'fstat ( fileno (bc->fp),&s ) != -1'
failed.
cvs update: warning: unrecognized response
'Abort' from cvs server
cvs [update aborted]: end of file from server
(consult above message if any)
But nevertheless the checked out file seems to be OK, because cvs
shows that as 'up to date' in my workspace. I have only one valid revision (1.1) in the repository.
So it seems, that the size of the file was ~GB
also in the first revision (sorry my fault). But however, i was able to checkin the first revision, but not the second one.

Frank Heitbrock <frankheitbrock>
Thu 25 May 2006 04:30:29 PM UTC, comment #1: 

I assume you have previously committed multiple revisions of the failing file successfully?  How big is the failing archive file (,v) in the repository?  CVS has a bad habit of loading an entire archive into memory for processing.  Even if you have committed a single 1GB revision on your server in the past, your failing file may be over 2GB or whatever your current process memory limit is. 

We have some code that already tries to use mmap for reads, in the hopes that the file system can handle some of the work for us, but it is not used for writes.  Can you check out revision 1.1 of this archive successfully?  How about the tip of a branch from this archive?  (The contrib/validate_repo script distributed with CVS 1.12.x may be useful for verifying this archive.)

If you don't appear to be having problems with the intensive read operations on this file, then mmap is working as designed for reads.  In the past, providing portable code for writable mmaps has proved troublesome, but perhaps it is worth the effort.

Derek Robert Price <dprice>
Group administrator
Fri 19 May 2006 03:14:53 PM UTC, original submission:  

When i try to commit a big pattern file modification (~100MB) into the cvs-repository, i get the following message.
"cvs [server aborted]: out of memory; can not allocate
1004771977 bytes"

If i try it a second time, i get exactly the same message with exactly the same number of bytes (also not dependent from the machine from which i try it)
The connection from the clients to the repository server is via ssh.
Strange is, that it was possible to add this file to the repository the first time. Also strange is, that i could before also add a very big pattern file with more than 1GB.


Frank Heitbrock <frankheitbrock>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

Only logged-in users can vote.

 

Follow 3 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2006-06-14 dprice StatusNeed Info Confirmed
2006-05-25 dprice Severity3 - Normal 1 - Wish
    StatusNone Need Info

Back to the top

Powered by Savane 3.13-4448.
Corresponding source code