bugGluster - Bugs: bug #26402, poor stripe write performance with...

 
 

bug #26402: poor stripe write performance with one subvolume

Submitted by:  Erick Tryzelaar <erickt>
Submitted on:  Wed 29 Apr 2009 10:23:09 PM UTC  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Improper behaviour
Status: PostponedPrivacy: Public
Assigned to: NoneOriginator Name: -
Open/Closed: ClosedRelease: 2.0.0git
Operating System: GNU/LinuxReproducibility: Every Time

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 25 Jun 2009 06:55:47 AM UTC, comment #3:

This bug has been moved to http://bugs.gluster.com

Please track the progress of the bug at http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=67

--
Gowda

Basavanagowda Kanur <gowda>
Project Member
Fri 01 May 2009 11:19:00 PM UTC, comment #2:

I believe it was with just the default arguments, which I believe is a stripe-size of 128KB. How can I check what the write-chunk-size is? Here's my server.vol file:

volume posix
type storage/posix
option directory /tmp/gluster
end-volume

volume locks
type features/locks
subvolumes posix
end-volume

volume io-threads
type performance/io-threads
option thread-count 16
subvolumes locks
end-volume

volume server
type protocol/server
option transport-type tcp
option auth.addr.io-threads.allow *
subvolumes io-threads
end-volume

and my client.vol file:

volume machine01
type protocol/client
option transport-type tcp
option remote-host machine01
option remote-subvolume io-threads
end-volume

volume stripe
type cluster/stripe
subvolumes machine01
end-volume

If I add a write-behind, the writes go up to 113MB/s:

volume write-behind
type performance/write-behind
option cache-size 1MB
# subvolumes stripe
subvolumes read-ahead
end-volume

I've also compared different block sizes and their read-write performance, without any of the client performance translators:

| block size | write | read |
| 1GB | 62MB/s | 113MB/s |
| 128MB | 62MB/s | 113MB/s |
| 64MB | 64MB/s | 113MB/s |
| 32MB | 60MB/s | 113MB/s |
| 16MB | 55MB/s | 113MB/s |
| 8MB | 50MB/s | 113MB/s |
| 4MB | 45MB/s | 113MB/s |
| 1MB | 23MB/s | 113MB/s |
| 512KB | 12MB/s | 113MB/s |
| 256KB | 8MB/s | 113MB/s |
| 128KB | 4MB/s | 113MB/s |
| 64KB | 2MB/s | 113MB/s |
| default | 4MB/s | 113MB/s |

So it appears that these performance issues really start happening around a block size of 16-32MB. I'm guessing that's still a much larger size than the write-chunk-size, and would correlate with some other performance problems I've seen with a stripe of 4 machines. With that, I'm seeing the overhead of striping limit the maximum amount a client can read unless the stripe-size is greater than 16MB. I'll run some numbers and file another bug about that.

Erick Tryzelaar <erickt>
Thu 30 Apr 2009 04:45:21 AM UTC, comment #1:

what is the stripe size being used and the chunk sizes of writes? With (stripe-size < write-chunk-size) and only one suvolume, there can be degradation in performance since a single write will be split into multiple chunks which are serially handled in the server.

Also, since optimal stripe size should enable each write to be split into equal chunks to be stored in subvolumes, it makes sense to set the stripe size equal to the write chunk size.

Raghavendra <raghavendra>
Project Member
Wed 29 Apr 2009 10:23:09 PM UTC, original submission:

I was debugging some poor performance issues with stripe across multiple machines, and I found a particularly bad edge case. I added a stripe volume, but with only one subvolume. This resulted in some terrible performance. If I just directly connect to the server, I can pull off about 50-70MB/s, but when I add the null-stripe, the performance drops to between 3-5MB/s. Reads, however, reach 112MB/s, which was the same we get without striping.

Erick Tryzelaar <erickt>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gowda (Posted a comment)
  • -unavailable- added by raghavendra (Posted a comment)
  • -unavailable- added by erickt (Submitted the item)
  • -unavailable- added by erickt
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 25 Jun 2009 06:55:47 AM UTCgowdaStatusNone=>Postponed
      Open/ClosedOpen=>Closed
    Thu 30 Apr 2009 04:45:21 AM UTCraghavendraOriginator Name=>-
    Wed 29 Apr 2009 10:23:09 PM UTCericktCarbon-Copy-=>Added erickt

    Back to the top


    Powered by Savane 3.1-cleanup1