bugpyFormex - Bugs: bug #39950, Memory allocation issue

 
 

bug #39950: Memory allocation issue

Submitted by:  francesco <francio>
Submitted on:  Thu 05 Sep 2013 09:59:57 AM UTC  
 
Category: RenderingSeverity: 4 - Important
Item Group: Functionality errorStatus: Fixed
Privacy: PublicAssigned to: Benedict Verhegghe <bverheg>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 29 Jan 2014 02:20:47 PM UTC, comment #7:

Builtin attributes have been removed from shaders.
Can now close the bug.

Benedict Verhegghe <bverheg>
Project AdministratorIn charge of this item.
Wed 30 Oct 2013 04:04:37 PM UTC, comment #6:

The memory leak in the new opengl engine seems to be restricted to the usage of builtin attributes. The attached file shows a comparison of using builtin=1 or builtin=0. Only the first case systematically leaks memory.

As the builtin shader attributes will disappear anyhow from newer versions of OpenGL, and are also not available in OpenGL/ES (Android devices!), I think we can decide here to remove the option alltogether. The GPU shader code will become cleaner as a result.

(file #29508)

Benedict Verhegghe <bverheg>
Project AdministratorIn charge of this item.
Wed 02 Oct 2013 03:24:50 PM UTC, comment #5:

The issue is fixed for the master branch in commit 4560b6cb.
See these results below: the drawing still takes up around 69 MB, but the total a mount of memory used does no longer increase
after each draw. The used memory is freed when the Actor is
destroyed.

Note that the issue has not been solved yet in the new opengl engine (in the opengl branch), so this bug is left open until
that one has been fixed too.

Benedict Verhegghe <bverheg>
Project AdministratorIn charge of this item.
Wed 02 Oct 2013 02:59:48 PM UTC, comment #4:

Actually it looks like the problem is the display lists, which are never really deleted.

And probably in the new engine, the same thing happens, but with the VBO objects.

Benedict Verhegghe <bverheg>
Project AdministratorIn charge of this item.
Wed 02 Oct 2013 02:26:49 PM UTC, comment #3:

Here is the attachment I forgot in my previous comment.

(file #29283)

Benedict Verhegghe <bverheg>
Project AdministratorIn charge of this item.
Wed 02 Oct 2013 02:25:34 PM UTC, comment #2:

Tracking the amount of memory used by a Python application is a complex task. The amount of free memory is not a good measure, because Linux will try to use up all available memory for buffers and cache. The amount of used memory (excluding buffers/cache) is more accurate, but it will still include the memory of all applications. So you should close as many other running applications as possible.

Then Python does its own garbage collection of objects that can no longer be referenced (the user does not free the memory himself, he merely just destroys references to the memory, and when no reference is left and the user can no longer access it, Python can free the memory). The garbage collection can be triggered by gb.collect().

I repeated Wouter's tests reporting the memory used, and doing a gb.collect() before each measurement. From the results in attachment, you can see that there is indeed a problem.
For a mesh with 1M triangles, the draw() instruction seems to
use up another 69M of memory. In the attachment you see the amount of memory used before and after the draw, the amount used and blocked by the draw, and the total amount of free memory.
The memory leak is confirmed by the second execution during the same session. Shutting down pyFormex free the memory.

It is important that we find out what is going on here, because the new GL engine even uses up more memory, but in a less consistent way. This also suggest that it is not related to the
display lists, as the new engine does not use these.

I guess it has to do with OpenGL buffers and the connection to
it through QT4. But we will have to install some memory tracker to find out.

I raised the severity.

Benedict Verhegghe <bverheg>
Project AdministratorIn charge of this item.
Wed 02 Oct 2013 09:16:47 AM UTC, comment #1:

I was just about to submit the same/similar problem:
For me the problem is that the memory does not get released after a clear(), following a draw().

Consider following example, where I keep drawing the same trisurface.

I get as output (my stl file is 25 MB in this case)

So basically the memory of something that was drawn, never gets released.

d = draw() / undraw(d) gives the same result.

Wouter Devriendt <whdvrien>
Project Member
Thu 05 Sep 2013 09:59:57 AM UTC, original submission:

I often get this error

Traceback (most recent call last):
File "/home/francio/phd2/prepoststudy/stent/cristalloSTL/FI_Cristallo31089.py", line 1941, in <module>
exit()
File "/home/francio/pyformex/pyformex/script.py", line 639, in exit
raise _Exit # ask exit from script only
_Exit
Traceback (most recent call last):
File "/home/francio/pyformex/pyformex/gui/draw.py", line 1536, in play

runAny(refresh=refresh)
File "/home/francio/pyformex/pyformex/script.py", line 610, in runAny

return runScript(appname,argv)
File "/home/francio/pyformex/pyformex/script.py", line 496, in runScript

res = playScript(file(fn,'r'),fn,fn,argv,pye)
File "/home/francio/pyformex/pyformex/script.py", line 366, in playScript

pf.debug("MemUsed = %s; vmSize = %s" % (memUsed(),vmSize()),pf.DEBUG.MEM)
File "/home/francio/pyformex/pyformex/script.py", line 738, in memUsed

return utils.memory_report()['MemUsed']
File "/home/francio/pyformex/pyformex/utils.py", line 1137, in memory_report

sta,out,err = system('cat /proc/meminfo')
File "/home/francio/pyformex/pyformex/utils.py", line 577, in system

P = Popen(cmd,shell=True,stdout=PIPE,stderr=PIPE)
File "/usr/lib/python2.7/subprocess.py", line 679, in _init_

errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1133, in _execute_child

self.pid = os.fork()
OSError
:
[Errno 12] Cannot allocate memory

normally when I run multiple times a code which creates a lot of variables stored in memory. so I have to restart pyformex. Is there a way to clean the global variables? I ve used forgetAll() but it does not solve the problem

francesco <francio>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #29508:  mem_free_builtin_0.txt added by bverheg (4KiB - text/plain)
file #29283:  mem_free_3.log added by bverheg (3KiB - text/x-log)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by bverheg (Posted a comment)
  • -unavailable- added by whdvrien (Posted a comment)
  • -unavailable- added by francio (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 10 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 29 Jan 2014 02:20:47 PM UTCbverhegOpen/ClosedOpen=>Closed
    Wed 30 Oct 2013 04:04:37 PM UTCbverhegAttached File-=>Added mem_free_builtin_0.txt, #29508
      StatusIn Progress=>Fixed
    Wed 02 Oct 2013 03:24:50 PM UTCbverhegStatusConfirmed=>In Progress
      Assigned toNone=>bverheg
    Wed 02 Oct 2013 02:26:49 PM UTCbverhegAttached File-=>Added mem_free_3.log, #29283
    Wed 02 Oct 2013 02:25:34 PM UTCbverhegCategoryNone=>Rendering
      Severity3 - Normal=>4 - Important
      Item GroupFeature request=>Functionality error
      StatusNone=>Confirmed

    Back to the top


    Powered by Savane 3.1-cleanup1