bugGluster - Bugs: bug #21596, system crash during stress test

 
 

bug #21596: system crash during stress test

Submitted by:  Sorin Badea <sorinbadea>
Submitted on:  Fri 16 Nov 2007 07:06:53 AM UTC  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Crash
Status: FixedPrivacy: Public
Assigned to: Amar Tumballi <amarts>Originator Name: 
Open/Closed: ClosedRelease: 
Operating System: GNU/LinuxReproducibility: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 12 Mar 2008 04:41:08 PM UTC, comment #24:

This bug is reported to be fixed in 1.3.8preX versions.

Amar Tumballi <amarts>
Project MemberIn charge of this item.
Thu 17 Jan 2008 05:28:16 AM UTC, comment #23:

fixed? as in whole thing is working now? or what? shall i close the bug? or if bug still exists, I can login and see. (I checked that the login details given by you is not working right now).

Amar Tumballi <amarts>
Project MemberIn charge of this item.
Thu 10 Jan 2008 08:39:44 AM UTC, comment #22:

hello,
fixed now.

Sorin

Sorin Badea <sorinbadea>
Thu 10 Jan 2008 06:09:01 AM UTC, comment #21:

hello,

Access to the machine running the glusterfs client:
ssh htuser@86.123.253.20 -p 6008 (mks6000)
I tried to start glustrefs as a non root user using SUID, but fuse
complains, if you need root credentials let me know.
in /etc/init.d there is a small script which stop/start glustrefs
with the required mounting point, logs, etc...

my Yahoo id is sbagabriel.

I can make another test with glusterfs runned from gdb.

Sorin

Sorin Badea <sorinbadea>
Tue 08 Jan 2008 03:01:37 PM UTC, comment #20:

Hello,

How can I get the patch-629 ?
I tried with the last version of 2.5 mainline
"tla get -A -unavailable- glusterfs--mainline--2.5 glusterfs-2.5"
but after 30 minutes of run the problem persists, glusterfs crash.
The configuration remains the same, 2 servers in afr and one client.

gdb backtrace:
(gdb) backtrace
#0 0xbfffe410 in __kernel_vsyscall ()
#1 0xb7e76fc0 in raise () from /lib/i686/libc.so.6
#2 0xb7e78851 in abort () from /lib/i686/libc.so.6
#3 0xb7eac41b in __fsetlocking () from /lib/i686/libc.so.6
#4 0xb7eb21a6 in malloc_usable_size () from /lib/i686/libc.so.6
#5 0xb7eb43cb in free () from /lib/i686/libc.so.6
#6 0xb7eb5d85 in malloc () from /lib/i686/libc.so.6
#7 0xb7fcb946 in gf_block_unserialize_transport (trans=0x8092fd0, max_block_size=268435456) at protocol.c:344
#8 0xb764616e in notify (this=0x8056448, event=2, data=0x8092fd0) at client-protocol.c:4952
#9 0xb7fcbbc7 in transport_notify (this=0x8092fd0, event=1) at transport.c:154
#10 0xb7fcc907 in sys_epoll_iteration (ctx=0xbffbae34) at epoll.c:54
#11 0xb7fcbcfc in poll_iteration (ctx=0xbffbae34) at transport.c:302
#12 0x0804a7e4 in main (argc=8, argv=0xbffbaf34) at glusterfs.c:401

glusterfs log:
2008-01-08 13:34:58 W [client-protocol.c:288:client_protocol_xfer] brick1: attempting to pipeline request type(0) op(34) with handshake
2008-01-08 13:34:58 W [client-protocol.c:288:client_protocol_xfer] brick2: attempting to pipeline request type(0) op(34) with handshake
2008-01-08 13:34:58 W [client-protocol.c:288:client_protocol_xfer] brick2: attempting to pipeline request type(0) op(34) with handshake
2008-01-08 13:34:58 W [client-protocol.c:288:client_protocol_xfer] brick2: attempting to pipeline request type(0) op(34) with handshake
2008-01-08 13:34:58 W [fuse-bridge.c:368:fuse_entry_cbk] glusterfs-fuse: 2: / => 1 Rehashing 0/0

---------
got signal (6), printing backtrace
---------
[0xbfffe420]
/lib/i686/libc.so.6(abort+0x101)[0xb7e78851]
/lib/i686/libc.so.6[0xb7eac41b]
/lib/i686/libc.so.6[0xb7eb21a6]
/lib/i686/libc.so.6[0xb7eb43cb]
/lib/i686/libc.so.6(malloc+0x85)[0xb7eb5d85]
/usr/local/lib/libglusterfs.so.0(gf_block_unserialize_transport+0x486)[0xb7fcb946]
/usr/local/lib/glusterfs/1.3.8/xlator/protocol/client.so(notify+0x29e)[0xb764616e]
/usr/local/lib/libglusterfs.so.0(transport_notify+0x37)[0xb7fcbbc7]
/usr/local/lib/libglusterfs.so.0(sys_epoll_iteration+0xd7)[0xb7fcc907]
/usr/local/lib/libglusterfs.so.0(poll_iteration+0x7c)[0xb7fcbcfc]
[glusterfs](main+0x434)[0x804a7e4]
/lib/i686/libc.so.6(__libc_start_main+0xdc)[0xb7e6475c]
[glusterfs][0x804a321]

Sorin Badea <sorinbadea>
Tue 08 Jan 2008 06:46:44 AM UTC, comment #19:

hello,

I fixed the problem with the permission but the problems occures again. My Yahoo Messanger ID is sbagabriel, so we can talk on yahoo.
meanwhile I'll setup again the test environment

Sorin

Sorin Badea <sorinbadea>
Mon 07 Jan 2008 04:51:08 PM UTC, comment #18:

Sorin,
as mentioned in amar's last message. the logs suggest that the permission of the export directory on brick1 is not set right. can you confirm that? also could you come on our IRC channel so that we can discuss easier?

avati

Anand Avati <avati>
Project Member
Mon 07 Jan 2008 03:13:01 PM UTC, comment #17:

Hello,

Do you have any news about the described problem ?
We are closing to the date of the "in production" delivery,
we also have invested a lot of time in the GlusterFS industrialization.

thanks,
Sorin

Sorin Badea <sorinbadea>
Tue 18 Dec 2007 10:07:11 AM UTC, comment #16:

Hello,
According with the gdb backtrace the glustrefs crashes in a
"calloc" call ( "#6 0xb7eca8b7 in fuse_ll_process (data=0x80556e0, buf=0x8099770 "@", len=64, ch=0x80556b8) at fuse_lowlevel.c:1135 )
I had a look in the calloc code, namely lib.c

it says:
void * calloc (size_t nmemb, size_t size)
113 {
114 return malloc(nmemb * size);
115 }
116
117
118 /**
119 * The presence of this global variable without any protected access
120 * to it explains why the "malloc/calloc" functions below are
121 * MT-unsafe !
122 */
123 static void * malloc_heap_top = NULL;
124 void * malloc (size_t size)
125 {
126 void * retval;
127
128 if (size <= 0)
129 return NULL;
130
131 /* Align on a 4B boundary */
132 size = ((size-1) & ~3) + 4;
133
134 if (! kernel_heap_top)
135 kernel_heap_top = _sos_brk(0);
136
137 if (! malloc_heap_top)
138 malloc_heap_top = kernel_heap_top;
139
140 retval = malloc_heap_top;
141 malloc_heap_top += size;
142
143 _sos_brk(malloc_heap_top);
144 return retval;
145 }

could be this the reason for the crash ?

thanks,
Sorin

Sorin Badea <sorinbadea>
Wed 12 Dec 2007 04:44:30 PM UTC, comment #15:

Thanks for the output file. I will check it again. In the mean time, can you check the permissions of directories in the brick1 (ie, server192.168.3.11), its saying 'permission denied' error.

Amar Tumballi <amarts>
Project MemberIn charge of this item.
Wed 12 Dec 2007 02:33:33 PM UTC, comment #14:

Hello,

I recompiled fuse as you suggested ( with kernel support ),
this time the glusterfs did not crash the whole system, however
it still crash.
It seems that the problem is in fuse, I attached logs and compilation directives in the file "for_zresearch file". maybe you'll see some wrong practice.

thanks,
Sorin

(file #14607)

Sorin Badea <sorinbadea>
Mon 10 Dec 2007 10:10:02 AM UTC, comment #13:

Hi Sorin,
I did built both packages as you below.

FUSE: (fuse-2.7.1)
./configure --enable-kernel-module --prefix=/usr

GlusterFS: (glusterfs-tla594)
./configure --disable-ibverbs --prefix=

uname -r = (2.6.21.5)

Now after going through your postings thoroughly again, I see that '--disable-kernel-module' option for fuse may be the reason, your client is crashing.

Try with using kernel module from the same source you have built the libfuse. (ie, use 'rmmod fuse.ko' and 'modprobe fuse' after installing new fuse module before running glusterfs).

Amar Tumballi <amarts>
Project MemberIn charge of this item.
Thu 06 Dec 2007 01:01:39 PM UTC, comment #12:

Hello,
During one test the targetDirectory was 1,4 Gbytes, for 321 files
During a second test targetDirectory was 8.4 Mbytes, for 3 files.
I noticed that the more files there are the most likley the glusterfs crashes.
Could you please send me the GlusterFS client and servers configuration,
OS version, kernel version, also how do you configure GlusterFS and Fuse before make and make install.

thanks,
Sorin

Sorin Badea <sorinbadea>
Thu 06 Dec 2007 12:47:21 PM UTC, comment #11:

Hi Sorin,
I am running the tests using your test tools. But it seems to be running fine for me. Btw, can I know, howmuch data do you have in export dir? ie, size of '$targetdirectory' ? That would help me to reproduce it sooner I think.

Regards,
Amar

Amar Tumballi <amarts>
Project MemberIn charge of this item.
Tue 04 Dec 2007 12:14:07 PM UTC, comment #10:

Hi,
One more thing:
We are using GlusterFS in one of our project and we experience severe crashes in a pre-production environment, as described in my previous message.
For us it is CRITICAL to get a solution asap on this issue. Please inform me on a tentative date for delivering a patch or a workaround.

Thank you,
Sorin Badea
Alcatel-Lucent

Sorin Badea <sorinbadea>
Tue 04 Dec 2007 11:46:57 AM UTC, comment #9:

Hello,
do you manage to reproduce the problem ?

I installed the same system on Fedora Core 6, it behave
in the same way, crash aftre about 15 minutes.

Sorin

Sorin Badea <sorinbadea>
Thu 22 Nov 2007 12:16:03 PM UTC, comment #8:

sorry abou the kernel is 2.6.17-5mdv

Sorin Badea <sorinbadea>
Thu 22 Nov 2007 12:14:19 PM UTC, comment #7:

ello,

I'm doing two kind of tests:
One using my own script.
Second using the Mercury tool, this tool is simulating
http requests against the website. The website is similar with Youtube. As media resources it mounts a GlusterFS file-system.

Yes, I have only one client and two servers in replication.
I attached the load and stress kit test ( testing.tools.tar )
I attached also the GlusterFS client and servers configuration from the last test.

Test scenario:
Test 1:
2 x startTest.pl instances, 30 users, 1 h, test OK.

Test 2:
Mercury load tests. Mercury is simulating http requests
against the website. The mercury tests loaded the machine
at about 0,5 therefore I launch a startTest.pl instance.
After about 15 minutes the system crashes.

My concern is that not only GlusterFC client crash but
the whole system is down, only ping through machine is working.

I speciffied my kernel version 2.6.7-5mdv ( Mandriva 2007 )

Fuse congigure command:
./configure --disable-kernel-module
Fuse version 2.7.1

GlusterFS client configure line:
./configure --sysconfdir=/etc --disable-server --disable-ibverbs

(file #14462, file #14463, file #14464, file #14465)

Sorin Badea <sorinbadea>
Thu 22 Nov 2007 09:57:40 AM UTC, comment #6:

Hi Sorin,
I am investigating the problem. The logs are not much helpful. I just wanted to know your set up. Just two servers and 1 client? or anything more? also which application did you run (or any script) to do the stress test? These information would help me to simulate the same setup locally.

Regards,
Amar

Amar Tumballi <amarts>
Project MemberIn charge of this item.
Mon 19 Nov 2007 01:22:04 PM UTC, comment #5:

Hello,
I tried with the flusterfs--mainline-2.5
I noticed a change in the AFR syntax translator,
( no need for option replicate *:n line )
Unfortunately the system crashes again, the system response
only to ping, no core file and only the Power Up/down can bring up
the machine.

Sorin Badea <sorinbadea>
Mon 19 Nov 2007 05:23:40 AM UTC, comment #4:

Please use GlusterFS with latest source. Refer following link to get the latest source.

http://gluster.org/docs/index.php/GlusterFS_FAQ#How_do_I_checkout_the_source.3F

Do let us know if it solves your problems.

Amar Tumballi <amarts>
Project MemberIn charge of this item.
Fri 16 Nov 2007 12:59:40 PM UTC, comment #3:

still crash during stress tests even with glusterfs-1.3.7.
I'm using Mandriva 2007 OS, kernel 2.6.17-5mdv.

thanks,
Sorin

Sorin Badea <sorinbadea>
Fri 16 Nov 2007 07:41:19 AM UTC, comment #2:

meaning version 1.3.7 ?

thanks,
Sorin

Sorin Badea <sorinbadea>
Fri 16 Nov 2007 07:15:46 AM UTC, comment #1:

Please upgrade to the latest release (or TLA checkout) which has a lot of stability fixes since 1.3.5.

Anand Avati <avati>
Project Member
Fri 16 Nov 2007 07:06:53 AM UTC, original submission:

Hello,
I'm using glustrefs as a filesystem for a website.
softeware: glusterfs 1.3.5
fuse 2.7.1
I configured two gluster servers in replication using the afr translator. I also use iothreads.
In normal condition the system works fine,
but during load simulation the system running glusterfs client
crashes and only the Power up/down button can help. The gluster servers works fine, no problem with them.
As hardware running the gluster client I have a hp dle380 G5 2x dual core CPU Intel.
the glusterfs client and server configuration are attached.
No core file is generated.
Any ideea about this ?

thnaks,
Sorin Badea

portion from glustrefs log file just before a crash

2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(529), lru=32/1024
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(529), lru=31/1024
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(800), lru=32/1024
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(460), lru=31/1024
2007-11-14 10:19:50 D [fuse-bridge.c:542:fuse_getattr] glusterfs-fuse: GETATTR 460 (/Picture/0_upl_1193823808716_img_top_rated_thumbnail.jpg)
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(808), lru=30/1024
2007-11-14 10:19:50 D [fuse-bridge.c:423:fuse_lookup] glusterfs-fuse: LOOKUP 123/0_upl_1194440688087_img_thumbnail.jpg (/Picture/0_upl_1194440688087_img_thumbnail.jpg)
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(460), lru=31/1024
2007-11-14 10:19:50 D [fuse-bridge.c:346:fuse_entry_cbk] glusterfs-fuse: ENTRY => 808
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(808), lru=32/1024
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(460), lru=31/1024
2007-11-14 10:19:50 D [read-ahead.c:334:read_ahead] readahead: RA at offset=0
2007-11-14 10:19:50 D [read-ahead.c:455:ra_readv] readahead: NEW REQ at offset=0 for size=8192
2007-11-14 10:19:50 D [read-ahead.c:467:ra_readv] readahead: received expected offset (0) when page_count=1
2007-11-14 10:19:50 D [read-ahead.c:394:dispatch_requests] readahead: IN-TRANSIT at offset=0.
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(808), lru=30/1024
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(340), lru=29/1024
2007-11-14 10:19:50 D [fuse-bridge.c:542:fuse_getattr] glusterfs-fuse: GETATTR 340 (/Picture/0_upl_1193300537816_img_top_rated_thumbnail.jpg)
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(397), lru=28/1024
2007-11-14 10:19:50 D [fuse-bridge.c:423:fuse_lookup] glusterfs-fuse: LOOKUP 397/1024_upl_1194263597072.jpg (/icons/1024_upl_1194263597072.jpg)
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(529), lru=29/1024

2007-11-14 13:10:20 E [fuse-bridge.c:1990:fuse_transport_init] glusterfs-fuse: fuse_mount failed (No such file or directory)

Sorin Badea <sorinbadea>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #14607:  for_zresearch added by sorinbadea (7KiB - application/octet-stream)
file #14462:  glusterfs-client.vol added by sorinbadea (1KiB - application/octet-stream)
file #14463:  glusterfs-server.vol added by sorinbadea (1KiB - application/octet-stream)
file #14464:  glusterfs-server2.vol added by sorinbadea (1KiB - application/octet-stream)
file #14465:  testing.tools.tar added by sorinbadea (10KiB - application/x-tar)
file #14410:  server1.vol added by sorinbadea (1KiB - application/octet-stream - glusterfs client and server files)
file #14411:  glusterfs-client.vol added by sorinbadea (1KiB - application/octet-stream - glusterfs client and server files)
file #14412:  server2.vol added by sorinbadea (1KiB - application/octet-stream - glusterfs client and server files)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by amarts (Posted a comment)
  • -unavailable- added by avati (Posted a comment)
  • -unavailable- added by sorinbadea (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 11 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 12 Mar 2008 04:41:08 PM UTCamartsStatusNone=>Fixed
      Open/ClosedOpen=>Closed
    Wed 12 Dec 2007 02:33:33 PM UTCsorinbadeaAttached File-=>Added for_zresearch, #14607
    Thu 22 Nov 2007 12:14:19 PM UTCsorinbadeaAttached File-=>Added glusterfs-client.vol, #14462
      Attached File-=>Added glusterfs-server.vol, #14463
      Attached File-=>Added glusterfs-server2.vol, #14464
      Attached File-=>Added testing.tools.tar, #14465
    Thu 22 Nov 2007 09:57:40 AM UTCamartsAssigned toNone=>amarts
    Fri 16 Nov 2007 07:06:53 AM UTCsorinbadeaAttached File-=>Added server1.vol, #14410
      Attached File-=>Added glusterfs-client.vol, #14411
      Attached File-=>Added server2.vol, #14412

    Back to the top


    Powered by Savane 3.1-cleanup1