bugGluster - Bugs: bug #25982, Strange AFR behaviour

 
 

bug #25982: Strange AFR behaviour

Submitted by:  Mathijs Krullaars <mathijsk>
Submitted on:  Tue 24 Mar 2009 04:23:20 AM UTC  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Improper behaviour
Status: PostponedPrivacy: Public
Assigned to: NoneOriginator Name: vikas
Open/Closed: ClosedRelease: 2.0-rc1-rc6
Operating System: GNU/LinuxReproducibility: Every Time

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 24 Jun 2009 06:26:55 PM UTC, comment #11:

This bug has been moved to http://bugs.gluster.com

Please track the progress of the bug at http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=51

--
Gowda

Basavanagowda Kanur <gowda>
Project Member
Thu 09 Apr 2009 02:27:12 PM UTC, comment #10:

A few fixes to the self-heal algorithm have gone in to the repository which should fix this problem. Can you please check with the latest git (or pre36)?

Vikas Gorur <vikasgp>
Project Member
Mon 30 Mar 2009 09:53:10 PM UTC, comment #9:

Vikas,
I just noticed something else very interesting. The original poster said that he is using Gluster across XFS volumes. I am doing the same thing, although he is using server side replication and I am using client side. Not sure if it matters, but it is an interesting fact.

Michael Taggart <mikeytag>
Mon 30 Mar 2009 08:56:42 PM UTC, comment #8:

Thanks Vikas,
I am running rc7 and here is a snippet of the log from one of my webservers:

2009-03-30 13:55:31 W [dht-common.c:244:dht_revalidate_cbk] distribute: mismatching layouts for <dir>
2009-03-30 13:55:31 W [fuse-bridge.c:301:need_fresh_lookup] fuse-bridge: revalidate of <dir> failed (Stale NFS file handle)
2009-03-30 13:55:32 W [dht-common.c:244:dht_revalidate_cbk] distribute: mismatching layouts for <dir>
2009-03-30 13:55:32 W [fuse-bridge.c:301:need_fresh_lookup] fuse-bridge: revalidate of <dir> failed (Stale NFS file handle)

Michael Taggart <mikeytag>
Mon 30 Mar 2009 12:30:04 PM UTC, comment #7:

RC7 has a major fix in replicate that solves many of the spurious 'split-brain' situations. Could you please check with RC7?

Vikas Gorur <vikasgp>
Project Member
Mon 30 Mar 2009 03:34:27 AM UTC, comment #6:

I agree. I wouldn't touch NFS with a 10 foot pole on a production setup. (I have had some horrible experiences in the past that leave me biased). What is funny is I am NOT using NFS at all. Just the bug in the glusterfs.log complains about a stale NFS handle, whatever that is.

The favorite-child problem seems like a big issue. I can't just have my servers pick one child all the time because it is not always the same server that has the right copy. When I notice this happen I usually see that the file is blank on one storage node and fully there on another node. Which one is anyone's guess based on who went down first etc.

Michael Taggart <mikeytag>
Sat 28 Mar 2009 03:21:15 PM UTC, comment #5:

Yes i had the same experience yesterday when i realized one of my servers was not running and was trying to sync it later with ls -lR.
It created a 0KB file and then gave me this in log:
Unable to resolve conflicting data of **. Please resolve manually by deleting the file ** from all but the preferred subvolume. Please consider 'option favorite-child <>'

For the Stale NFS thing, first read this: http://www.gluster.org/docs/index.php/GlusterFS_cookbook#Exporting_over_NFS

I would not use nfs at all.

Mathijs Krullaars <mathijsk>
Fri 27 Mar 2009 09:12:22 PM UTC, comment #4:

FWIW, I am also getting these errors:

2009-03-27 13:58:24 W [dht-common.c:244:dht_revalidate_cbk] distribute: mismatching layouts for <directory>

and

2009-03-27 14:10:18 W [fuse-bridge.c:301:need_fresh_lookup] fuse-bridge: revalidate of <directory> (Stale NFS file handle)

For the stale problem, I found an old post where a dev said to enable option lookup-unhashed yes and then do a stat on the files from the client. So I did:

find /san/ -exec stat {} \;

However, this doesn't seem to have remedied the errors. Not sure if this is even related to this bug. Do you think I should run stat on the server instead of the client?

Michael Taggart <mikeytag>
Fri 27 Mar 2009 09:07:41 PM UTC, comment #3:

Mathijs,
I am also seeing this on my rc7 structure. I have 4 bricks with distribute and replicate. I am replicating bricks 1 and 2, 3 and 4 and then distributing the replications.

I seem to fill up my logs with the dreaded "can't replicate file please consider option favorite-child" if I bring down a brick for maintenance and then bring it back up. I will have to say that it seems that rc7 is getting better in this regard, but I am still seeing problems now and then.
Mike

Michael Taggart <mikeytag>
Tue 24 Mar 2009 06:49:23 PM UTC, comment #2:

Checked some more and in a split brain situation the behaviour is not perfect yet, the server that gets the file accessed first will replicate it to the other one, no matter if it is the latest or not.

Mathijs Krullaars <mathijsk>
Tue 24 Mar 2009 06:18:51 PM UTC, comment #1:

I just checked rc7, and it seems this issue is solved.

Mathijs Krullaars <mathijsk>
Tue 24 Mar 2009 04:23:20 AM UTC, original submission:

I use GlusterFS with server side replication on XFS volumes.
The test setup is on 2 servers, they are also client, so exported directory's are also mounted with gluster client locally.

When i echo a string to a test file, take one server down, change the string, bring the server back up and cat the file, it will not always be the the latest string.

When i test this with rc1 to rc4, the contents of the file will revert to the wrong string, or it will even be corrupted.
When i test this with rc6, the file is the good version when i cat it on a folder that is connected to the server with the right file, and the file is updated on the other server, so everything is like it should be.

But when i cat it first on a client that is connected to the server that was taken down, it will return the old string and update the contents of the newer file with the old one.

When all servers are up all the time this will work ok, but when one would go down the results can be unexpected.

Mathijs Krullaars <mathijsk>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #17788:  volume-files added by mathijsk (2KiB - application/octet-stream - My volume files)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gowda (Posted a comment)
  • -unavailable- added by vikasgp (Posted a comment)
  • -unavailable- added by mikeytag (Posted a comment)
  • -unavailable- added by mathijsk (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 24 Jun 2009 06:26:55 PM UTCgowdaStatusNone=>Postponed
      Open/ClosedOpen=>Closed
    Mon 30 Mar 2009 12:30:04 PM UTCvikasgpOriginator Name=>vikas
    Tue 24 Mar 2009 04:23:20 AM UTCmathijskAttached File-=>Added volume-files, #17788

    Back to the top


    Powered by Savane 3.1-cleanup1