bugGluster - Bugs: bug #26054, deadlock in fetch_spec when...

 
 

bug #26054: deadlock in fetch_spec when mounting gluster

Submitted by:  Erick Tryzelaar <erickt>
Submitted on:  Mon 30 Mar 2009 10:00:47 PM UTC  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Crash
Status: PostponedPrivacy: Public
Assigned to: Raghavendra <raghavendra>Originator Name: erickt
Open/Closed: ClosedRelease: 2.0.0rc7
Operating System: GNU/LinuxReproducibility: Every Time

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 25 Jun 2009 06:51:05 AM UTC, comment #4:

This bug has been moved to http://bugs.gluster.com

Please track the progress of the bug at http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=66

--
Gowda

Basavanagowda Kanur <gowda>
Project Member
Wed 01 Apr 2009 07:34:17 AM UTC, comment #3:

You were correct in your diagnosis. A fix is on its way to repository.

Raghavendra <raghavendra>
Project MemberIn charge of this item.
Tue 31 Mar 2009 10:29:35 PM UTC, comment #2:

I've figured out how this happened. We configured automount through an NIS server, with a line like this:

gluster -fstype=glusterfs :/etc/glusterfs/client.vol

Then we tried to go to the mountpoint without /etc/glusterfs/client.vol existing. I'm guessing that since the file didn't exist, gluster interpreted "/etc/glusterfs/client.vol" as a hostname, and specified the "--volfile-server=/etc/glusterfs/client.vol". When the file does exist, /usr/sbin/gluster is started with "--volfile=/etc/glusterfs/client.vol" and mount correctly. It's repeatable in that if we later remove the client.vol file and restart gluster, it'll hang again.

Erick Tryzelaar <erickt>
Tue 31 Mar 2009 06:18:28 AM UTC, comment #1:

--volfile-server option should specify a server. If the volume specfication is present locally, it can be either specified using -f option or --volfile-server should be passed localhost and a glusterfs server should be running locally.

Raghavendra <raghavendra>
Project MemberIn charge of this item.
Mon 30 Mar 2009 10:00:47 PM UTC, original submission:

We've got a autofs-loaded gluster that's deadlocking while trying to mount a gluster fs. We got this stack trace:

#0 0x00002aaaaaf37766 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaabeb43d in event_dispatch_epoll (event_pool=0x607320) at event.c:830
#2 0x00000000004046b4 in fetch_spec (ctx=0x607010) at fetch-spec.c:200
#3 0x00000000004032da in main (argc=5, argv=0x7fffa3520ce8) at glusterfsd.c:289

However, the spec file is local, not remote. It was mounted with:

/bin/mount -t glusterfs -s -o rw,intr /etc/glusterfs/client.vol /mnt/gluster

Which spawned off this:

/usr/sbin/glusterfs --log-level=WARNING --volfile-server=/etc/glusterfs/client.vol --volfile-server-port=6996 /mnt/gluster

I'm not sure if you need this, but the ctx that used in fetch_spec is this according to gdb:

$2 = {cmd_args = {volfile_server = 0x100000001 <Address 0x100000001 out of bounds>, volume_file = 0x0, log_level = GF_LOG_NONE,
log_file = 0x60735000000000 <Address 0x60735000000000 out of bounds>, volfile_server_port = 0, volfile_server_transport = 0x0,
pid_file = 0x0, no_daemon_mode = 0, run_id = 0x9ea648a000000000 <Address 0x9ea648a000000000 out of bounds>, debug_mode = -1,
xlator_options = {next = 0xa0862aa0ffffffff, prev = 0xa19af790ffffffff}, fuse_direct_io_mode_flag = -1,
volfile_check = -880207200, fuse_entry_timeout = -4.9620051038398657e+87, fuse_attribute_timeout = -6.8227563537816586e+88,
volume_name = 0xd6fe7420ffffffff <Address 0xd6fe7420ffffffff out of bounds>, non_local = -1,
icon_name = 0xdafed1a0ffffffff <Address 0xdafed1a0ffffffff out of bounds>, fuse_nodev = -1, fuse_nosuid = -608137200,
mount_point = 0xdcdeb3a0ffffffff <Address 0xdcdeb3a0ffffffff out of bounds>,
volfile_id = 0xdda9ac90ffffffff <Address 0xdda9ac90ffffffff out of bounds>},
process_uuid = 0xdebe95a0ffffffff <Address 0xdebe95a0ffffffff out of bounds>, specfp = 0xdf898e90ffffffff,
pidfp = 0xe09e77a0ffffffff, fin = -1 '', timer = 0xe27e59a0ffffffff, ib = 0xe3495290ffffffff, pool = 0xe45e3ba0ffffffff,
graph = 0xe5293490ffffffff, top = 0xe6475820ffffffff, event_pool = 0xe7125110ffffffff, lock = {__data = {__lock = -1,
__count = 3894884896, __owner = -1, __nusers = 3908186896, __kind = -1, __spins = -368632800, __list = {
__prev = 0xead21510ffffffff, __next = 0xebe6fe20ffffffff}}, __size = " :'�\0203� \034\a�\020\025� �",
__align = -1718340819410223105}, xl_count = -1, volfile_checksum = 3971086096}

Erick Tryzelaar <erickt>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #17840:  client.vol added by erickt (14KiB - application/octet-stream - server and client specs)
file #17841:  server.vol added by erickt (614B - application/octet-stream - server and client specs)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gowda (Posted a comment)
  • -unavailable- added by raghavendra (Posted a comment)
  • -unavailable- added by erickt (Submitted the item)
  • -unavailable- added by erickt
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 10 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 25 Jun 2009 06:51:05 AM UTCgowdaStatusIn Progress=>Postponed
      Open/ClosedOpen=>Closed
    Tue 31 Mar 2009 07:10:02 AM UTCraghavendraReproducibilityIntermittent=>Every Time
    Tue 31 Mar 2009 07:09:45 AM UTCraghavendraStatusInvalid=>In Progress
    Tue 31 Mar 2009 06:18:28 AM UTCraghavendraStatusNone=>Invalid
      Assigned toNone=>raghavendra
      Originator Name=>erickt
    Mon 30 Mar 2009 10:00:47 PM UTCericktAttached File-=>Added client.vol, #17840
      Attached File-=>Added server.vol, #17841
      Carbon-Copy-=>Added erickt

    Back to the top


    Powered by Savane 3.1-cleanup1