bugGluster - Bugs: bug #26626, Inodes never freed + valgrind...

 
 

bug #26626: Inodes never freed + valgrind report

Submitted by:  Krzysztof Strasburger <strasbur>
Submitted on:  Wed 20 May 2009 06:28:22 AM UTC  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Improper behaviour
Status: PostponedPrivacy: Public
Assigned to: Raghavendra <raghavendra>Originator Name: -
Open/Closed: ClosedRelease: 2.0.1
Operating System: GNU/LinuxReproducibility: Every Time

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 25 Jun 2009 06:22:15 AM UTC, comment #3:

The valgrind report is showing a huge 'still reachable' memory blocks because glusterfs does not destroy the data structures that are still in-use while umount is done.

These 'still reachable' reports should go away, once we have even more graceful
shutdown of glusterfs.

This bug has been moved to http://bugs.gluster.com

Please track the progress of the bug at http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=61

--
Gowda

Basavanagowda Kanur <gowda>
Project Member
Tue 23 Jun 2009 07:30:14 AM UTC, comment #2:

Amar Tumballi advised me to set drop_caches to 3, but it did not help.
I understand that it is not a memory leak, as valgrind claims that the pointers are not forgotten. Repeating the same operation does not cause additional memory allocations, so the inodes are cached and used, as needed. However, it is not good to see all your memory consumed forever, only because somebody ran du on a big directory tree.
My attachement is still accessible, I tried even to download it, to be sure.
You can download it directly via http://savannah.nongnu.org/bugs/download.php?file_id=18169

Krzysztof Strasburger <strasbur>
Mon 22 Jun 2009 05:57:23 PM UTC, comment #1:

Hi,

Inodes are freed when kernel sends forget on them. In other words the extent inodes are cached in glusterfs depends on kernel. Hence you are seeing huge memory consumption, but it's not memory leak. doing echo 3 > /proc/sys/vm/drop_caches should bring down memory consumption.

Btw, I did not find any attachment. Am I missing anything?

regards,
Raghavendra.

Raghavendra <raghavendra>
Project MemberIn charge of this item.
Wed 20 May 2009 06:28:22 AM UTC, original submission:

Running anything, that scans the directory tree recursively (du, ls -R), leads to increasing memory consumption by the glusterfs client. Here is the most remarkable part of the report from running glusterfs client under valgrind --leak-check=full --show-reachable=yes:

==15979== 104,480,064 bytes in 251,154 blocks are still reachable in loss record
73 of 73
==15979== at 0x68EEC77: calloc (in /usr/local/lib/valgrind/x86-linux/vgpreloa
d_memcheck.so)
==15979== by 0x6915D22: __inode_create (inode.c:450)
==15979== by 0x6915D7C: inode_new (inode.c:466)
==15979== by 0x6B1B2A1: fuse_lookup (fuse-bridge.c:441)
==15979== by 0x6B3CEE7: do_lookup (fuse_lowlevel.c:444)
==15979== by 0x6B3E9DD: fuse_ll_process (fuse_lowlevel.c:1182)
==15979== by 0x6B403AA: fuse_session_process (fuse_session.c:90)
==15979== by 0x6B232BF: fuse_thread_proc (fuse-bridge.c:2486)
==15979== by 0x6942A57: pthread_start_thread (in /lib/libpthread-0.10.so)
==15979== by 0x6A2E2E9: clone (in /lib/libc-2.3.6.so)

I have no idea, whether the error roots are in glusterfs or libfuse, but anyway, glusterfs depends on fuse.
Full report is attached as compressed file.
If you need more tests, feel free to contact me.

Krzysztof Strasburger <strasbur>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #18169:  glusterfs.valgrind.gz added by strasbur (6KiB - application/gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gowda (Posted a comment)
  • -unavailable- added by raghavendra (Posted a comment)
  • -unavailable- added by strasbur (Submitted the item)
  • -unavailable- added by strasbur
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 25 Jun 2009 06:22:15 AM UTCgowdaStatusNone=>Postponed
      Open/ClosedOpen=>Closed
    Mon 22 Jun 2009 05:57:23 PM UTCraghavendraAssigned toNone=>raghavendra
      Originator Name=>-
    Wed 20 May 2009 06:28:22 AM UTCstrasburAttached File-=>Added glusterfs.valgrind.gz, #18169
      Carbon-Copy-=>Added raghavendra

    Back to the top


    Powered by Savane 3.1-cleanup1