bugGluster - Bugs: bug #24551, Gluster and dovecot files...

 
 

bug #24551: Gluster and dovecot files corruption

Submitted by:  None
Submitted on:  Mon 13 Oct 2008 08:23:04 PM UTC  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Improper behaviour
Status: FixedPrivacy: Public
Assigned to: NoneOriginator Name: Anand Avati
Originator Email: -unavailable-Open/Closed: Closed
Release: Operating System: GNU/Linux
Reproducibility: Every Time

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 23 Jun 2009 01:49:24 PM UTC, comment #11:

Assumed as fixed, as per the discussion in the thread.

--
Gowda

Basavanagowda Kanur <gowda>
Project Member
Thu 14 May 2009 11:41:35 AM UTC, comment #10:

This seems to be fixed with 2.0.1 (ie. no error from dovecot nor gluster).

Anonymous
Tue 03 Feb 2009 10:58:17 AM UTC, comment #9:

Can you check the behavior with 2.0.0rc1 release? If fixed in that release, I will be closing the bug.

Proper path construction on server side should fix this bug.

Regards,

Amar Tumballi <amarts>
Project Member
Sat 25 Oct 2008 09:05:21 PM UTC, comment #8:

Well, not entirely.

I went ahead and added performance translators to the client:

volume writeback
type performance/write-behind
option aggregate-size 4MB
option flush-behind on
subvolumes client
end-volume

volume readahead
type performance/read-ahead
option page-size 1MB # unit in bytes
option page-count 2 # cache per file = (page-count x page-size)
subvolumes writeback
end-volume

volume iocache
type performance/io-cache
option force-revalidate-timeout 3
option page-size 512KB
option cache-size 167772160
subvolumes readahead
end-volume

Now I get, from the server:

2008-10-25 22:57:54 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 22:57:56 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 23:00:58 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:00:58 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:00 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:00 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:04 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 23:01:05 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 23:01:08 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:08 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:09 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:09 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:10 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:10 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:11 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 23:01:13 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:13 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:14 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 23:01:16 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:16 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:17 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:17 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:18 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:18 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:21 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:21 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:25 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 23:01:35 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 23:01:36 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:36 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:38 E [server-protocol.c:2780:server_stub_resume] server: stat returning ENOENT: -1 (2)
2008-10-25 23:01:42 E [server-protocol.c:2717:server_stub_resume] server: open returning ENOENT: -1 (2)
2008-10-25 23:01:42 E [fd.c:411:fd_unref] fd.c: fd is NULL
2008-10-25 23:01:44 E [posix.c:619:posix_releasedir] brick: pfd->dir is NULL for fd=0x936c1a0 path=<NULL
[repeating]

And from the client:
2008-10-25 22:57:54 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 13163: OPEN() / => -1 (Is a directory)
2008-10-25 22:57:55 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 13853: OPEN() / => -1 (Is a directory)
2008-10-25 23:00:58 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 21592: OPEN() /Maildir/dovecot.index.log => -1 (No such file or directory)
2008-10-25 23:01:00 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 23267: OPEN() /Maildir/dovecot.index.cache => -1 (No such file or directory)
2008-10-25 23:01:02 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 25019: OPEN() / => -1 (Is a directory)
2008-10-25 23:01:03 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 25857: OPEN() / => -1 (Is a directory)
2008-10-25 23:01:04 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 27345: OPEN() / => -1 (Is a directory)
2008-10-25 23:01:06 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 29202: OPEN() / => -1 (Is a directory)
2008-10-25 23:01:06 E [fuse-bridge.c:633:fuse_fd_cbk] glusterfs-fuse: 29343: OPEN() / => -1 (Is a directory)
[...]

glusterfs--mainline--3.0--patch-471

Anonymous
Sat 25 Oct 2008 08:20:36 PM UTC, comment #7:

Thanks, it works like a charm now.

Anonymous
Thu 23 Oct 2008 04:52:19 AM UTC, comment #6:

Please check with patch-448. There is a fix for a bug which was treating fd as a 32bit value instead of 64.

Anand Avati <avati>
Project Member
Wed 22 Oct 2008 09:36:48 PM UTC, comment #5:

With glusterfs--mainline--3.0--patch-447 I get:

E [server-protocol.c:4760:server_fsyncdir] server: unresolved fd 0
from the server process and:

E [fuse-bridge.c:869:fuse_err_cbk] glusterfs-fuse: 4116: FSYNCDIR() ERR => -1 (Invalid argument)
from the client.

I will provide any info you ask for.

Anonymous
Wed 15 Oct 2008 11:50:13 AM UTC, comment #4:

Yes, I'm testing it on the same machine.

ls -l /usr/local/sbin/glusterfs*
-rwxr-xr-x 1 root root 75714 2008-10-14 21:29 /usr/local/sbin/glusterfs
lrwxrwxrwx 1 root root 9 2008-10-14 21:29 /usr/local/sbin/glusterfsd -> glusterfs

/usr/local/sbin/glusterfsd --versionglusterfs 1.4.0pre6 built on Oct 14 2008 21:28:43
Repository revision: glusterfs--mainline--3.0--patch-429

Anonymous
Wed 15 Oct 2008 03:47:55 AM UTC, comment #3:

Have you upgraded both the client and the server to the latest TLA revision? Please double check.

avati

Anand Avati <avati>
Project Member
Tue 14 Oct 2008 07:50:41 PM UTC, comment #2:

Thanks, I tested it out and it works mostly, but it still gives me a bunch of these after a while:

server-protocol.c:4633:server_fsyncdir] server: unresolved fd 0

Anonymous
Tue 14 Oct 2008 09:40:30 AM UTC, comment #1:

Fix is in glusterfs--mainline--3.0--patch-423

Anand Avati <avati>
Project Member
Mon 13 Oct 2008 08:23:04 PM UTC, original submission:

It seems that dovecot doesn't like gluster.
I'm using a very simple setup to track this down, with just storage/posix, features/posix-locks on the server and
a protocol/client. Glusterfs 1.4.pre5, but 1.3.12 corrupts my files as well.
Using postal, an SMTP benchmark and dovecot LDA I'm getting these errors and dovecot's indexes[0] corruption pretty soon:
piano deliver(user): fdatasync_path(/mnt/Maildir/new) failed: Invalid argument
Oct 13 21:51:52 piano deliver(user): msgid=<48F3A6D8.34D.001.user>: save failed to INBOX: Timeout while waiting for lock
Oct 13 21:51:52 piano postfix/cleanup[15340]: 9451A36467A: message-id=<48F3A6D8.6C7.001.user>
[...]
stat(/mnt/Maildir/tmp) failed: Transport endpoint is not connected
[...]
Corrupted transaction log file /mnt/Maildir/dovecot.index.log: duplicate transaction log sequence (2)
Oct 13 21:56:40 piano deliver(user): open() failed with index cache file /mnt/Maildir/dovecot.index.cache: File descriptor in bad state
Oct 13 21:56:40 piano deliver(user): close() failed with file /mnt/Maildir/dovecot.index.log: File descriptor in bad state
Oct 13 21:56:40 piano deliver(user): pwrite() failed with file /mnt/Maildir/dovecot.index.log: File descriptor in bad state
The relevant dovecot configuration:
mail_nfs_storage = yes
mail_nfs_index = yes
dotlock_use_excl = yes
mmap_disable = yes
(More informations here: http://wiki.dovecot.org/MailLocation/SharedDisk)
The file attached shows an error:
2008-10-13 21:51:45 E [fd.c:207:gf_fd_fdptr_get] fd: invalid argument
2008-10-13 21:51:45 E [server-protocol.c:4024:server_fsyncdir] server: unresolved fd -1
which I think is caused by fd_no in server-protocol.c always being -1, but I don't know if it's the cause.
[0] http://wiki.dovecot.org/IndexFiles

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #16668:  gluster-dovecot.log.gz added by None (80KiB - application/x-gzip - Log)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gowda (Posted a comment)
  • -unavailable- added by amarts (Posted a comment)
  • -unavailable- added by avati (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 23 Jun 2009 01:49:24 PM UTCgowdaStatusNone=>Fixed
      Open/ClosedOpen=>Closed
    Tue 14 Oct 2008 09:40:30 AM UTCavatiOriginator Name=>Anand Avati
      Originator Email=>-unavailable-
    Mon 13 Oct 2008 08:23:04 PM UTCNoneAttached File-=>Added gluster-dovecot.log.gz, #16668

    Back to the top


    Powered by Savane 3.1-cleanup1