bugGluster - Bugs: bug #26392, afr round robining broken in git

 
 

bug #26392: afr round robining broken in git

Submitted by:  Erick Tryzelaar <erickt>
Submitted on:  Wed 29 Apr 2009 04:31:04 AM UTC  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Improper behaviour
Status: InvalidPrivacy: Public
Assigned to: NoneOriginator Name: Anand Avati
Open/Closed: ClosedRelease: 2.0.0git
Operating System: GNU/LinuxReproducibility: Every Time

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 25 Jun 2009 06:00:54 AM UTC, comment #5:

cluster/replicate's read load-balancing policy change is intentional and has been moved away from round-robin.

this is not a bug.

please mail to -unavailable- for assistance over configuration issues.

--
Gowda

Basavanagowda Kanur <gowda>
Project Member
Wed 29 Apr 2009 07:39:23 AM UTC, comment #4:

(arg, savannah ate my message). So we could have a separate read-subvolume for the clients, but since we'd be using this across 1000+ clients the scale would get challenging. Our current idea for the servers would have 4 machines striped, then 4 of those mirrored, then those blocks of 16 machines distributed N times. We could write a script to generate the .vol files which wouldn't be so bad and just hope that we get pretty evenly distributed reads across those machines.

If I remember correctly from our meeting, doesn't gluster maintain one channel for metadata and another for data? Is there any way we could have a similar pattern for this? One machine "owns" the metadata, and all the accesses must go to that one machine, but any of the machines in the AFR could serve the reads? That'd probably do a much better job of distributing the load than statically defining it in the .vol. Especially if you use a random seed to pick the node to read from.

The ideal setup (I would imagine) would be for the clients to ask the server how busy they each were (or have the servers tell the clients how busy they are), and then just copy that file from the idle-est machine. That's probably not something that can be easily done in the short term though.

Erick Tryzelaar <erickt>
Wed 29 Apr 2009 06:00:32 AM UTC, comment #3:

Do you think assigning different option read-subvolume to different clients might be a suitable workaround?

Anand Avati <avati>
Project Member
Wed 29 Apr 2009 05:55:41 AM UTC, comment #2:

Thanks Anand. So is this behavior going to stay? I'm not sure how likely this would happen in a production setting, but in my testing the gluster clients always chose the same server to copy a file from So file A could be copied from machine X for all clients. This meant that the total aggregate bandwidth available to copy that file was 1 divided by the number of clients. Since we're building a write-once, read-many style caching cluster, keeping file attribute times consistent isn't really that important to us. Since we expect many clients to be accessing the same files, we really would want to be to parallelize the load across all the machines in the mirror.

Erick Tryzelaar <erickt>
Wed 29 Apr 2009 05:32:54 AM UTC, comment #1:

The commit you mentioned changes the behavior (in an expected way). The round robin in replicate assigns subvolumes in a round robing manner to different files looked up. Once a subvolume is assigned, all further reads happen from the same subvolume for that file. This is necessary to keep file attribute times consistent.

I have verified that the latest git at least, different files do indeed get load balanced among subvolumes, and readv() calls go to different servers.

Anand Avati <avati>
Project Member
Wed 29 Apr 2009 04:31:04 AM UTC, original submission:

In 2.0.0rc7, gluster would round-robin read requests across all the nodes in a replicate brick. This broke in 2.0.0rc8 and above though. I've managed to track down the git commit that broke this feature:

d8bc2834d056a23f55bd86a5fd704bd81e35836d

The commit previously successfully round robins request, and this one sends the request to only one machine, the last one listed in the subvolumes list.

Erick Tryzelaar <erickt>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gowda (Posted a comment)
  • -unavailable- added by avati (Posted a comment)
  • -unavailable- added by erickt (Submitted the item)
  • -unavailable- added by erickt
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 25 Jun 2009 06:00:54 AM UTCgowdaStatusNone=>Invalid
      Open/ClosedOpen=>Closed
    Wed 29 Apr 2009 05:32:54 AM UTCavatiOriginator Name=>Anand Avati
    Wed 29 Apr 2009 04:31:04 AM UTCericktCarbon-Copy-=>Added erickt

    Back to the top


    Powered by Savane 3.1-cleanup1