bugGluster - Bugs: bug #26776, Using cluster/replicate as a...

 
 

bug #26776: Using cluster/replicate as a cluster/unify namespace brick crashes

Submitted by:  Jonathan Steffan <damaestro>
Submitted on:  Wed 10 Jun 2009 05:14:11 PM UTC  
Votes:  5  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Crash
Status: PostponedPrivacy: Public
Assigned to: NoneOriginator Name: -
Open/Closed: ClosedRelease: 2.0.2
Operating System: GNU/LinuxReproducibility: Every Time

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 25 Jun 2009 06:42:16 AM UTC, comment #6:

This bug has been moved to http://bugs.gluster.com

Please track the progress of the bug at http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=64

--
Gowda

Basavanagowda Kanur <gowda>
Project Member
Thu 18 Jun 2009 03:32:15 PM UTC, comment #5:

We have removed the cluster/unify translator and are just going with multiple mounts to segment content. Everything is working now so I suspect this is an issue with running cluster/replicate or cluster/distribute under the unify translator.. or it's an issue with the switch scheduler, which I have opened another bug for.

Jonathan Steffan <damaestro>
Tue 16 Jun 2009 06:04:27 PM UTC, comment #4:

http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=22

Jonathan Steffan <damaestro>
Tue 16 Jun 2009 04:27:36 PM UTC, comment #3:
Jonathan Steffan <damaestro>
Tue 16 Jun 2009 04:21:57 PM UTC, comment #2:

Okay, this looks like it's less an issue about using these translators together and more about something crashing/leaking in the client. Using a replicated namespace brick just makes everything happen faster. The crash happens after a collection of the following is seen on the server side:

[2009-06-15 19:54:41] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /mpath/to/some/sort/of/content/20812.jpg/30 failed: Not a directory
[2009-06-15 19:55:05] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/real_estate.jpg/0 failed: Not a directory
[2009-06-15 19:55:08] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /mpath/to/some/sort/of/content/real_estate.jpg/600 failed: Not a directory
[2009-06-15 19:58:55] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/720music.jpg/oliveCOVER.jpg failed: Not a directory
[2009-06-15 19:59:09] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /mpath/to/some/sort/of/content/0/20812.jpg/30 failed: Not a directory
[2009-06-15 19:59:53] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/34971.jpg/400 failed: Not a directory
[2009-06-15 20:11:14] E [posix.c:270:posix_lookup] iops_lun_disk0: lstat on /path/to/some/sort/of/content/headlines failed: Not a directory
[..... many lstat .....]
[2009-06-15 20:11:15] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/headlines failed: Not a directory
[2009-06-15 20:28:38] E [posix.c:382:posix_opendir] unify_lun_disk0: opendir failed on /path/to/some/sort/of/other/content/bang_4.jpg: Not a directory
[2009-06-15 20:31:01] E [posix.c:1298:posix_utimens] iops_lun_disk0: utimes on /path/to/disk-ld1/path/to/some/sort/of/content/90/.112294.jpg.pBk95v failed: No such file or directory
[2009-06-15 21:19:12] E [posix.c:1147:posix_chmod] iops_lun_disk0: chmod on /path/to/some/sort/of/content/70/.279.jpg.XARvhk failed: No such file or directory
[2009-06-15 21:21:47] E [posix.c:1147:posix_chmod] iops_lun_disk0: chmod on /mpath/to/some/sort/of/content/80/.1180.jpg.49pYSO failed: No such file or directory
[2009-06-15 19:40:11] E [posix.c:382:posix_opendir] unify_lun_disk0: opendir failed on /mpath/to/some/sort/of/content/0/60/31068.jpg: Not a directory
[2009-06-15 19:54:41] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/20812.jpg/30 failed: Not a directory
[2009-06-15 19:55:05] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/real_estate.jpg/0 failed: Not a directory
[2009-06-15 19:55:08] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/real_estate.jpg/600 failed: Not a directory
[2009-06-15 19:58:55] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/720music.jpg/oliveCOVER.jpg failed: Not a directory
[2009-06-15 19:59:09] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/20812.jpg/30 failed: Not a directory
[2009-06-15 19:59:53] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/34971.jpg/400 failed: Not a directory
[2009-06-15 20:07:58] E [posix.c:1147:posix_chmod] iops_lun_disk0: chmod on /path/to/some/sort/of/content/.81445.jpg.gQiXuD failed: No such file or directory
[2009-06-15 20:11:15] E [posix.c:270:posix_lookup] unify_lun_disk0: lstat on /path/to/some/sort/of/content/headlines failed: Not a directory
[2009-06-15 20:26:12] E [posix.c:1147:posix_chmod] iops_lun_disk0: chmod on /mpath/to/some/sort/of/content/0/.105545.jpg.ZeY2ga failed: No such file or directory

This is looking like it's possible that the issue is actually with the switch scheduler screwing up.

volume main_storage
type cluster/unify
# unify_namespace is a cluster/replicate to two servers
option namespace unify_namespace
option scheduler switch
# Anything not defined here ends up in 'bulk'
option scheduler.switch.case jpg:iops;gif:iops;png:iops;flv:iops;swf:iops;css:iops;xml:iops;htm:iops;wav:bulkaudio
subvolumes iops bulkaudio bulk
end-volume

Jonathan Steffan <damaestro>
Wed 10 Jun 2009 11:38:32 PM UTC, comment #1:

This also happens when using a cluster/afr volume for the namespace brick of a cluster/unify.

Jonathan Steffan <damaestro>
Wed 10 Jun 2009 05:14:11 PM UTC, original submission:

When using a cluster/replicate brick for the namespace of cluster/unify crashes shortly after data population.

How to crash:

1.) Start populating data.
2.) for i in {1..1000}; do ls -R /path/to/glusterfsmount; done
3.) Wait for short period.
4.) Client disconnects from servers and the filesystem is left in an unusable state.

Jonathan Steffan <damaestro>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gowda (Posted a comment)
  • -unavailable- added by damaestro (Voted in favor of this item)
  • -unavailable- added by damaestro (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 5 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 25 Jun 2009 06:42:16 AM UTCgowdaStatusNone=>Postponed
      Originator Name=>-
      Open/ClosedOpen=>Closed
    Wed 10 Jun 2009 11:38:32 PM UTCdamaestroCarbon-Copy-=>Added damaestro

    Back to the top


    Powered by Savane 3.1-cleanup1