Wed 12 Mar 2008 04:41:08 PM UTC, comment #24:
This bug is reported to be fixed in 1.3.8preX versions.
|
Thu 17 Jan 2008 05:28:16 AM UTC, comment #23:
fixed? as in whole thing is working now? or what? shall i close the bug? or if bug still exists, I can login and see. (I checked that the login details given by you is not working right now).
|
Thu 10 Jan 2008 08:39:44 AM UTC, comment #22:
hello,
fixed now.
Sorin
|
Thu 10 Jan 2008 06:09:01 AM UTC, comment #21:
hello,
Access to the machine running the glusterfs client:
ssh htuser@86.123.253.20 -p 6008 (mks6000)
I tried to start glustrefs as a non root user using SUID, but fuse
complains, if you need root credentials let me know.
in /etc/init.d there is a small script which stop/start glustrefs
with the required mounting point, logs, etc...
my Yahoo id is sbagabriel.
I can make another test with glusterfs runned from gdb.
Sorin
|
Tue 08 Jan 2008 03:01:37 PM UTC, comment #20:
Hello,
How can I get the patch-629 ?
I tried with the last version of 2.5 mainline
"tla get -A -unavailable- glusterfs--mainline--2.5 glusterfs-2.5"
but after 30 minutes of run the problem persists, glusterfs crash.
The configuration remains the same, 2 servers in afr and one client.
gdb backtrace:
(gdb) backtrace
#0 0xbfffe410 in __kernel_vsyscall ()
#1 0xb7e76fc0 in raise () from /lib/i686/libc.so.6
#2 0xb7e78851 in abort () from /lib/i686/libc.so.6
#3 0xb7eac41b in __fsetlocking () from /lib/i686/libc.so.6
#4 0xb7eb21a6 in malloc_usable_size () from /lib/i686/libc.so.6
#5 0xb7eb43cb in free () from /lib/i686/libc.so.6
#6 0xb7eb5d85 in malloc () from /lib/i686/libc.so.6
#7 0xb7fcb946 in gf_block_unserialize_transport (trans=0x8092fd0, max_block_size=268435456) at protocol.c:344
#8 0xb764616e in notify (this=0x8056448, event=2, data=0x8092fd0) at client-protocol.c:4952
#9 0xb7fcbbc7 in transport_notify (this=0x8092fd0, event=1) at transport.c:154
#10 0xb7fcc907 in sys_epoll_iteration (ctx=0xbffbae34) at epoll.c:54
#11 0xb7fcbcfc in poll_iteration (ctx=0xbffbae34) at transport.c:302
#12 0x0804a7e4 in main (argc=8, argv=0xbffbaf34) at glusterfs.c:401
glusterfs log:
2008-01-08 13:34:58 W [client-protocol.c:288:client_protocol_xfer] brick1: attempting to pipeline request type(0) op(34) with handshake
2008-01-08 13:34:58 W [client-protocol.c:288:client_protocol_xfer] brick2: attempting to pipeline request type(0) op(34) with handshake
2008-01-08 13:34:58 W [client-protocol.c:288:client_protocol_xfer] brick2: attempting to pipeline request type(0) op(34) with handshake
2008-01-08 13:34:58 W [client-protocol.c:288:client_protocol_xfer] brick2: attempting to pipeline request type(0) op(34) with handshake
2008-01-08 13:34:58 W [fuse-bridge.c:368:fuse_entry_cbk] glusterfs-fuse: 2: / => 1 Rehashing 0/0
---------
got signal (6), printing backtrace
---------
[0xbfffe420]
/lib/i686/libc.so.6(abort+0x101)[0xb7e78851]
/lib/i686/libc.so.6[0xb7eac41b]
/lib/i686/libc.so.6[0xb7eb21a6]
/lib/i686/libc.so.6[0xb7eb43cb]
/lib/i686/libc.so.6(malloc+0x85)[0xb7eb5d85]
/usr/local/lib/libglusterfs.so.0(gf_block_unserialize_transport+0x486)[0xb7fcb946]
/usr/local/lib/glusterfs/1.3.8/xlator/protocol/client.so(notify+0x29e)[0xb764616e]
/usr/local/lib/libglusterfs.so.0(transport_notify+0x37)[0xb7fcbbc7]
/usr/local/lib/libglusterfs.so.0(sys_epoll_iteration+0xd7)[0xb7fcc907]
/usr/local/lib/libglusterfs.so.0(poll_iteration+0x7c)[0xb7fcbcfc]
[glusterfs](main+0x434)[0x804a7e4]
/lib/i686/libc.so.6(__libc_start_main+0xdc)[0xb7e6475c]
[glusterfs][0x804a321]
|
Tue 08 Jan 2008 06:46:44 AM UTC, comment #19:
hello,
I fixed the problem with the permission but the problems occures again. My Yahoo Messanger ID is sbagabriel, so we can talk on yahoo.
meanwhile I'll setup again the test environment
Sorin
|
Mon 07 Jan 2008 04:51:08 PM UTC, comment #18:
Sorin,
as mentioned in amar's last message. the logs suggest that the permission of the export directory on brick1 is not set right. can you confirm that? also could you come on our IRC channel so that we can discuss easier?
avati
|
Mon 07 Jan 2008 03:13:01 PM UTC, comment #17:
Hello,
Do you have any news about the described problem ?
We are closing to the date of the "in production" delivery,
we also have invested a lot of time in the GlusterFS industrialization.
thanks,
Sorin
|
Tue 18 Dec 2007 10:07:11 AM UTC, comment #16:
Hello,
According with the gdb backtrace the glustrefs crashes in a
"calloc" call ( "#6 0xb7eca8b7 in fuse_ll_process (data=0x80556e0, buf=0x8099770 "@", len=64, ch=0x80556b8) at fuse_lowlevel.c:1135 )
I had a look in the calloc code, namely lib.c
it says:
void * calloc (size_t nmemb, size_t size)
113 {
114 return malloc(nmemb * size);
115 }
116
117
118 /**
119 * The presence of this global variable without any protected access
120 * to it explains why the "malloc/calloc" functions below are
121 * MT-unsafe !
122 */
123 static void * malloc_heap_top = NULL;
124 void * malloc (size_t size)
125 {
126 void * retval;
127
128 if (size <= 0)
129 return NULL;
130
131 /* Align on a 4B boundary */
132 size = ((size-1) & ~3) + 4;
133
134 if (! kernel_heap_top)
135 kernel_heap_top = _sos_brk(0);
136
137 if (! malloc_heap_top)
138 malloc_heap_top = kernel_heap_top;
139
140 retval = malloc_heap_top;
141 malloc_heap_top += size;
142
143 _sos_brk(malloc_heap_top);
144 return retval;
145 }
could be this the reason for the crash ?
thanks,
Sorin
|
Wed 12 Dec 2007 04:44:30 PM UTC, comment #15:
Thanks for the output file. I will check it again. In the mean time, can you check the permissions of directories in the brick1 (ie, server192.168.3.11), its saying 'permission denied' error.
|
Wed 12 Dec 2007 02:33:33 PM UTC, comment #14:
Hello,
I recompiled fuse as you suggested ( with kernel support ),
this time the glusterfs did not crash the whole system, however
it still crash.
It seems that the problem is in fuse, I attached logs and compilation directives in the file "for_zresearch file". maybe you'll see some wrong practice.
thanks,
Sorin
(file #14607)
|
Mon 10 Dec 2007 10:10:02 AM UTC, comment #13:
Hi Sorin,
I did built both packages as you below.
FUSE: (fuse-2.7.1)
./configure --enable-kernel-module --prefix=/usr
GlusterFS: (glusterfs-tla594)
./configure --disable-ibverbs --prefix=
uname -r = (2.6.21.5)
Now after going through your postings thoroughly again, I see that '--disable-kernel-module' option for fuse may be the reason, your client is crashing.
Try with using kernel module from the same source you have built the libfuse. (ie, use 'rmmod fuse.ko' and 'modprobe fuse' after installing new fuse module before running glusterfs).
|
Thu 06 Dec 2007 01:01:39 PM UTC, comment #12:
Hello,
During one test the targetDirectory was 1,4 Gbytes, for 321 files
During a second test targetDirectory was 8.4 Mbytes, for 3 files.
I noticed that the more files there are the most likley the glusterfs crashes.
Could you please send me the GlusterFS client and servers configuration,
OS version, kernel version, also how do you configure GlusterFS and Fuse before make and make install.
thanks,
Sorin
|
Thu 06 Dec 2007 12:47:21 PM UTC, comment #11:
Hi Sorin,
I am running the tests using your test tools. But it seems to be running fine for me. Btw, can I know, howmuch data do you have in export dir? ie, size of '$targetdirectory' ? That would help me to reproduce it sooner I think.
Regards,
Amar
|
Tue 04 Dec 2007 12:14:07 PM UTC, comment #10:
Hi,
One more thing:
We are using GlusterFS in one of our project and we experience severe crashes in a pre-production environment, as described in my previous message.
For us it is CRITICAL to get a solution asap on this issue. Please inform me on a tentative date for delivering a patch or a workaround.
Thank you,
Sorin Badea
Alcatel-Lucent
|
Tue 04 Dec 2007 11:46:57 AM UTC, comment #9:
Hello,
do you manage to reproduce the problem ?
I installed the same system on Fedora Core 6, it behave
in the same way, crash aftre about 15 minutes.
Sorin
|
Thu 22 Nov 2007 12:16:03 PM UTC, comment #8:
sorry abou the kernel is 2.6.17-5mdv
|
Thu 22 Nov 2007 12:14:19 PM UTC, comment #7:
ello,
I'm doing two kind of tests:
One using my own script.
Second using the Mercury tool, this tool is simulating
http requests against the website. The website is similar with Youtube. As media resources it mounts a GlusterFS file-system.
Yes, I have only one client and two servers in replication.
I attached the load and stress kit test ( testing.tools.tar )
I attached also the GlusterFS client and servers configuration from the last test.
Test scenario:
Test 1:
2 x startTest.pl instances, 30 users, 1 h, test OK.
Test 2:
Mercury load tests. Mercury is simulating http requests
against the website. The mercury tests loaded the machine
at about 0,5 therefore I launch a startTest.pl instance.
After about 15 minutes the system crashes.
My concern is that not only GlusterFC client crash but
the whole system is down, only ping through machine is working.
I speciffied my kernel version 2.6.7-5mdv ( Mandriva 2007 )
Fuse congigure command:
./configure --disable-kernel-module
Fuse version 2.7.1
GlusterFS client configure line:
./configure --sysconfdir=/etc --disable-server --disable-ibverbs
(file #14462, file #14463, file #14464, file #14465)
|
Thu 22 Nov 2007 09:57:40 AM UTC, comment #6:
Hi Sorin,
I am investigating the problem. The logs are not much helpful. I just wanted to know your set up. Just two servers and 1 client? or anything more? also which application did you run (or any script) to do the stress test? These information would help me to simulate the same setup locally.
Regards,
Amar
|
Mon 19 Nov 2007 01:22:04 PM UTC, comment #5:
Hello,
I tried with the flusterfs--mainline-2.5
I noticed a change in the AFR syntax translator,
( no need for option replicate *:n line )
Unfortunately the system crashes again, the system response
only to ping, no core file and only the Power Up/down can bring up
the machine.
|
Mon 19 Nov 2007 05:23:40 AM UTC, comment #4:
Please use GlusterFS with latest source. Refer following link to get the latest source.
http://gluster.org/docs/index.php/GlusterFS_FAQ#How_do_I_checkout_the_source.3F
Do let us know if it solves your problems.
|
Fri 16 Nov 2007 12:59:40 PM UTC, comment #3:
still crash during stress tests even with glusterfs-1.3.7.
I'm using Mandriva 2007 OS, kernel 2.6.17-5mdv.
thanks,
Sorin
|
Fri 16 Nov 2007 07:41:19 AM UTC, comment #2:
meaning version 1.3.7 ?
thanks,
Sorin
|
Fri 16 Nov 2007 07:15:46 AM UTC, comment #1:
Please upgrade to the latest release (or TLA checkout) which has a lot of stability fixes since 1.3.5.
|
Fri 16 Nov 2007 07:06:53 AM UTC, original submission:
Hello,
I'm using glustrefs as a filesystem for a website.
softeware: glusterfs 1.3.5
fuse 2.7.1
I configured two gluster servers in replication using the afr translator. I also use iothreads.
In normal condition the system works fine,
but during load simulation the system running glusterfs client
crashes and only the Power up/down button can help. The gluster servers works fine, no problem with them.
As hardware running the gluster client I have a hp dle380 G5 2x dual core CPU Intel.
the glusterfs client and server configuration are attached.
No core file is generated.
Any ideea about this ?
thnaks,
Sorin Badea
portion from glustrefs log file just before a crash
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(529), lru=32/1024
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(529), lru=31/1024
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(800), lru=32/1024
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(460), lru=31/1024
2007-11-14 10:19:50 D [fuse-bridge.c:542:fuse_getattr] glusterfs-fuse: GETATTR 460 (/Picture/0_upl_1193823808716_img_top_rated_thumbnail.jpg)
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(808), lru=30/1024
2007-11-14 10:19:50 D [fuse-bridge.c:423:fuse_lookup] glusterfs-fuse: LOOKUP 123/0_upl_1194440688087_img_thumbnail.jpg (/Picture/0_upl_1194440688087_img_thumbnail.jpg)
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(460), lru=31/1024
2007-11-14 10:19:50 D [fuse-bridge.c:346:fuse_entry_cbk] glusterfs-fuse: ENTRY => 808
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(808), lru=32/1024
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(460), lru=31/1024
2007-11-14 10:19:50 D [read-ahead.c:334:read_ahead] readahead: RA at offset=0
2007-11-14 10:19:50 D [read-ahead.c:455:ra_readv] readahead: NEW REQ at offset=0 for size=8192
2007-11-14 10:19:50 D [read-ahead.c:467:ra_readv] readahead: received expected offset (0) when page_count=1
2007-11-14 10:19:50 D [read-ahead.c:394:dispatch_requests] readahead: IN-TRANSIT at offset=0.
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(808), lru=30/1024
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(340), lru=29/1024
2007-11-14 10:19:50 D [fuse-bridge.c:542:fuse_getattr] glusterfs-fuse: GETATTR 340 (/Picture/0_upl_1193300537816_img_top_rated_thumbnail.jpg)
2007-11-14 10:19:50 D [inode.c:351:__active_inode] fuse/inode: activating inode(397), lru=28/1024
2007-11-14 10:19:50 D [fuse-bridge.c:423:fuse_lookup] glusterfs-fuse: LOOKUP 397/1024_upl_1194263597072.jpg (/icons/1024_upl_1194263597072.jpg)
2007-11-14 10:19:50 D [inode.c:381:__passive_inode] fuse/inode: passivating inode(529), lru=29/1024
2007-11-14 13:10:20 E [fuse-bridge.c:1990:fuse_transport_init] glusterfs-fuse: fuse_mount failed (No such file or directory)
|