bugConcurrent Versions System - Bugs: bug #16630, out of memory message during commit

 
 

bug #16630: out of memory message during commit

Submitted by:  Frank Heitbrock <frankheitbrock>
Submitted on:  Fri 19 May 2006 03:14:53 PM UTC  
 
Category: Bug ReportSeverity: 1 - Wish
Item Group: NoneStatus: Confirmed
Privacy: PublicAssigned to: None
Open/Closed: OpenRelease: 
Fixed Release: NoneFixed Feature Release: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

 Spam posted by an anonymous
Wed 14 Jun 2006 05:34:47 PM UTC, comment #3:

Yes, that number is very close to the first one you reported (1,004,772,198 - 1,004,771,977 = 221 bytes, probably the number of bytes needed to preparse the archive header). There is nothing that can be done on this end about this, short of rewriting the way CVS allocates and uses memory when parsing a file, or possibly adding binary diff code to the CVS server to avoid creating such large archives in the first place (upping your server memory or refraining from checking in such large binary files might be a solution you could implement without touching the CVS internals). We would love to see a patch!

Derek Robert Price <dprice>
Project Administrator
Wed 14 Jun 2006 11:55:37 AM UTC, comment #2:

I have checked the size of the failing (,v) file in the archive today. The size is 1,004,772,198 bytes. I have renamed the failing file and checked it out again.
At the end of the new checkout, i got the follwoing error message from cvs:
"cvs:buffer.c:1384: stdio_buffer_shutdown:
Assertion 'fstat ( fileno (bc->fp),&s ) != -1'
failed.
cvs update: warning: unrecognized response
'Abort' from cvs server
cvs [update aborted]: end of file from server
(consult above message if any)
But nevertheless the checked out file seems to be OK, because cvs
shows that as 'up to date' in my workspace. I have only one valid revision (1.1) in the repository.
So it seems, that the size of the file was ~GB
also in the first revision (sorry my fault). But however, i was able to checkin the first revision, but not the second one.

Frank Heitbrock <frankheitbrock>
Thu 25 May 2006 04:30:29 PM UTC, comment #1:

I assume you have previously committed multiple revisions of the failing file successfully? How big is the failing archive file (,v) in the repository? CVS has a bad habit of loading an entire archive into memory for processing. Even if you have committed a single 1GB revision on your server in the past, your failing file may be over 2GB or whatever your current process memory limit is.

We have some code that already tries to use mmap for reads, in the hopes that the file system can handle some of the work for us, but it is not used for writes. Can you check out revision 1.1 of this archive successfully? How about the tip of a branch from this archive? (The contrib/validate_repo script distributed with CVS 1.12.x may be useful for verifying this archive.)

If you don't appear to be having problems with the intensive read operations on this file, then mmap is working as designed for reads. In the past, providing portable code for writable mmaps has proved troublesome, but perhaps it is worth the effort.

Derek Robert Price <dprice>
Project Administrator
Fri 19 May 2006 03:14:53 PM UTC, original submission:

When i try to commit a big pattern file modification (~100MB) into the cvs-repository, i get the following message.
"cvs [server aborted]: out of memory; can not allocate
1004771977 bytes"

If i try it a second time, i get exactly the same message with exactly the same number of bytes (also not dependent from the machine from which i try it)
The connection from the clients to the repository server is via ssh.
Strange is, that it was possible to add this file to the repository the first time. Also strange is, that i could before also add a very big pattern file with more than 1GB.

Frank Heitbrock <frankheitbrock>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can click here to add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

Please enter the title of George Orwell's famous dystopian book (it's a date):

 

 

Follow 3 latest changes.

Date Changed By Updated Field Previous Value => Replaced By
Wed 14 Jun 2006 05:34:47 PM UTCdpriceStatusNeed Info=>Confirmed
Thu 25 May 2006 04:30:29 PM UTCdpriceSeverity3 - Normal=>1 - Wish
  StatusNone=>Need Info

Back to the top


Powered by Savane 3.1-cleanup