Fri 07 Mar 2008 07:04:10 AM UTC, comment #3:
this issue seem to be because of linux page cache not re-issuing a fresh read for sections which could not be read in the previous shot but within a page.. what i mean is
app: read 1008 bytes -> kernel: read 4096 bytes -> returns 1008 bytes -> app gets 1008 bytes
app: read next 1008 bytes -> kernel thinks it knows all data till 4096 bytes, does not issue read to fuse/glusterfs to fill from 1008 to 4096.
this is happening because ofile is opened in write-only mode, and glusterfs enforces direct_io fops on this fd. this means that the page cache is bypassed. all writes are coming to glusterfs directly, whereas the ifile fd is coming through the page cache (which is not seeing the writes, and thinks that the small 1008 byte cache is not stale).
if you mount glusterfs with "-d DISABLE", then all fops go through page cache (it invalidates the cache appropriately on detecting writes) and the py program works fine.
this behaviour is similar to opening ofile fd with O_DIRECT and ifile without O_DIRECT on your local filesystem.
the closest fix i see is remounting with '-d DISABLE'. for fuse after linux-2.6.24 direct io flag may not be necessary since large writes (> 4KB) support for non-direct-io is being added into fuse. the purpose of direct io flag set by glusterfs is only to get larger write block size.
|