bugGluster - Bugs: bug #22760, glusterfsd forking issues under...

 
 

bug #22760: glusterfsd forking issues under OpenVZ

Submitted by:  christoph mallau <dx0r>
Submitted on:  Fri 28 Mar 2008 10:44:29 AM UTC  
 
Category: GlusterFSSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Improper behaviour
Status: NonePrivacy: Public
Assigned to: NoneOriginator Name: Amar
Open/Closed: ClosedRelease: 
Operating System: GNU/LinuxReproducibility: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 30 Oct 2008 10:33:34 AM UTC, comment #6:

hai frnd,
so u too havng the same problm na??
ok
i too had till now.. but i found a solution ryt now..
it may work.. try
my glustrfsd showd this error
"glusterfsd: WARNING: ignoring stale pidfile for PID 30852"

so i coudnt run or mount...

also der is no glustrfsd.pid file running showd by commnd "top"

thus chk /usr/local/var/run
der vil b a pid fie of glusterfsd

jst delete dis file..
den u can run glusterfsd vith out the abouv error of stale...
ok try man.....

:)chettaii:)

eldhose <chettai>
Fri 29 Aug 2008 08:14:47 PM UTC, comment #5:

bug timed out..

Please test with new release http://ftp.zresearch.com/pub/gluster/glusterfs/1.4-qa/glusterfs-1.4.0qa40.tar.gz and re open the bug if still exists.

Regards,

Amar Tumballi <amarts>
Project Member
Mon 14 Apr 2008 03:40:19 PM UTC, comment #4:

Hi Amar,

I applied the patches in glusterfs--mainline--2.5 via tla, as you suggested.

GlusterClient2 / # glusterfs --version
glusterfs 1.3.8 built on Apr 14 2008 15:55:29
Repository revision: glusterfs--mainline--2.5--patch-740
Copyright (c) 2006, 2007, 2008 Z RESEARCH Inc. <http://www.zresearch.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

It seems that the problem does not occur any more. Having several forks of glusterfs does not produce any warnings, and all forked instances are visible through a ps aux:

GlusterClient2 / # ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 3708 592 ? Ss 17:22 0:00 init [3]
root 20340 0.0 0.0 15132 728 ? Ss 17:22 0:00 /usr/sbin/syslog-ng
root 20367 0.0 0.0 12092 656 ? S 17:22 0:00 /usr/sbin/crond
root 21741 0.0 0.0 18440 612 ? Ss 17:22 0:00 vzctl: ttyp0
root 21742 0.0 0.1 16448 1764 ttyp0 Ss 17:22 0:00 -bash
root 21749 0.0 0.0 34836 780 ? Ssl 17:22 0:00 [glusterfs]
root 21753 0.0 0.1 48936 1608 ? Ssl 17:22 0:00 [glusterfs]

Unfortunately, since I am not authorized to give you access to our testing environment, I cannot give you the opportunity to investigate what the solution was, actually :(

If there is any kind of information you may benefit from, I will try my best to supply you with anything you need.

Best Regards,
Christoph Mallau

christoph mallau <dx0r>
Sat 12 Apr 2008 01:37:08 AM UTC, comment #3:

We will look into it further. But if you still have problems with this, with latest patch (glusterfs--mainline--2.5), let us know. Also it would be great if we can get a remote-access to your system so I can fix the bug very soon. If you need details for giving access, let me know.

amar (at) zresearch.com

Amar Tumballi <amarts>
Project Member
Mon 31 Mar 2008 07:57:38 AM UTC, comment #2:

Okay,

I cleaned the system from the previous installation of glusterfs (1.3.7) und compiled the 1.3.8pre4 sources.

Christophs_Teestube glusterfs-1.3.8pre4 # /usr/sbin/glusterfs --version
glusterfs 1.3.8pre4 built on Mar 31 2008 09:17:25
Repository revision: glusterfs--mainline--2.5--patch-719
Copyright (c) 2006, 2007, 2008 Z RESEARCH Inc. <http://www.zresearch.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.

I do not get any more warnings about a stale PID file at all,
but there are no additional forks of glusterfsd in my process list, too :(

I once again included an strace:

strace /etc/init.d/start_cluster
execve("/etc/init.d/start_cluster", ["/etc/init.d/start_cluster"], [/* 22 vars */]) = 0
brk(0) = 0x5c1000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bfb582000
uname({sys="Linux", node="Christophs_Teestube", ...}) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=19511, ...}) = 0
mmap(NULL, 19511, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b1bfb583000
close(3) = 0
open("/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\0\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=373768, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bfb588000
mmap(NULL, 1423704, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b1bfb589000
mprotect(0x2b1bfb5d6000, 1048576, PROT_NONE) = 0
mmap(0x2b1bfb6d6000, 57344, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4d000) = 0x2b1bfb6d6000
mmap(0x2b1bfb6e4000, 2392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2b1bfb6e4000
close(3) = 0
open("/lib/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=14512, ...}) = 0
mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b1bfb784000
mprotect(0x2b1bfb786000, 2097152, PROT_NONE) = 0
mmap(0x2b1bfb986000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x2b1bfb986000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\334\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1297552, ...}) = 0
mmap(NULL, 3404792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b1bfb988000
mprotect(0x2b1bfbabe000, 2097152, PROT_NONE) = 0
mmap(0x2b1bfbcbe000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x136000) = 0x2b1bfbcbe000
mmap(0x2b1bfbcc3000, 17400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2b1bfbcc3000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bfbcc8000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bfbcc9000
arch_prctl(ARCH_SET_FS, 0x2b1bfbcc8dd0) = 0
mprotect(0x2b1bfbcbe000, 16384, PROT_READ) = 0
mprotect(0x2b1bfb986000, 4096, PROT_READ) = 0
mprotect(0x2b1bfb782000, 4096, PROT_READ) = 0
munmap(0x2b1bfb583000, 19511) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/dev/tty", O_RDWR|O_NONBLOCK) = 3
close(3) = 0
brk(0) = 0x5c1000
brk(0x5e2000) = 0x5e2000
getuid() = 0
getgid() = 0
geteuid() = 0
getegid() = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/proc/meminfo", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bfb583000
read(3, "MemTotal: 2097152 kB\nMemFre"..., 1024) = 771
close(3) = 0
munmap(0x2b1bfb583000, 4096) = 0
rt_sigaction(SIGCHLD, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGCHLD, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_DFL}, 8) = 0
uname({sys="Linux", node="Christophs_Teestube", ...}) = 0
stat("/root/glusterfs-1.3.8pre4", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
getpid() = 21724
getppid() = 21723
socket(PF_FILE, SOCK_STREAM, 0) = 3
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0
socket(PF_FILE, SOCK_STREAM, 0) = 3
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0
open("/etc/nsswitch.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=508, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bfb583000
read(3, "# /etc/nsswitch.conf:\n# $Header:"..., 4096) = 508
read(3, "", 4096) = 0
close(3) = 0
munmap(0x2b1bfb583000, 4096) = 0
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=19511, ...}) = 0
mmap(NULL, 19511, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b1bfb583000
close(3) = 0
open("/lib/libnss_compat.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\23\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=31432, ...}) = 0
mmap(NULL, 2127080, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b1bfbcca000
mprotect(0x2b1bfbcd1000, 2093056, PROT_NONE) = 0
mmap(0x2b1bfbed0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x2b1bfbed0000
close(3) = 0
open("/lib/libnsl.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20A\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=84792, ...}) = 0
mmap(NULL, 2190032, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b1bfbed2000
mprotect(0x2b1bfbee6000, 2093056, PROT_NONE) = 0
mmap(0x2b1bfc0e5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13000) = 0x2b1bfc0e5000
mmap(0x2b1bfc0e7000, 6864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2b1bfc0e7000
close(3) = 0
mprotect(0x2b1bfc0e5000, 4096, PROT_READ) = 0
mprotect(0x2b1bfbed0000, 4096, PROT_READ) = 0
munmap(0x2b1bfb583000, 19511) = 0
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=19511, ...}) = 0
mmap(NULL, 19511, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b1bfb583000
close(3) = 0
open("/lib/libnss_nis.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220 \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=39288, ...}) = 0
mmap(NULL, 2135256, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b1bfc0e9000
mprotect(0x2b1bfc0f2000, 2093056, PROT_NONE) = 0
mmap(0x2b1bfc2f1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x2b1bfc2f1000
close(3) = 0
open("/lib/libnss_files.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0` \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=43336, ...}) = 0
mmap(NULL, 2139432, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b1bfc2f3000
mprotect(0x2b1bfc2fd000, 2093056, PROT_NONE) = 0
mmap(0x2b1bfc4fc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x2b1bfc4fc000
close(3) = 0
mprotect(0x2b1bfc4fc000, 4096, PROT_READ) = 0
mprotect(0x2b1bfc2f1000, 4096, PROT_READ) = 0
munmap(0x2b1bfb583000, 19511) = 0
open("/etc/passwd", O_RDONLY) = 3
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
lseek(3, 0, SEEK_CUR) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=1319, ...}) = 0
mmap(NULL, 1319, PROT_READ, MAP_SHARED, 3, 0) = 0x2b1bfb583000
lseek(3, 1319, SEEK_SET) = 1319
munmap(0x2b1bfb583000, 1319) = 0
close(3) = 0
getpgrp() = 21723
rt_sigaction(SIGCHLD, {0x436110, [], SA_RESTORER, 0x2b1bfb9b85d0}, {SIG_DFL}, 8) = 0
getrlimit(RLIMIT_NPROC, {rlim_cur=441024, rlim_max=441024}) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/etc/init.d/start_cluster", O_RDONLY) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffaf5407a0) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
read(3, "#!/bin/bash\n#\n##################"..., 80) = 80
lseek(3, 0, SEEK_SET) = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
dup2(3, 255) = 255
close(3) = 0
fcntl(255, F_SETFD, FD_CLOEXEC) = 0
fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(255, {st_mode=S_IFREG|0755, st_size=432, ...}) = 0
lseek(255, 0, SEEK_CUR) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "#!/bin/bash\n#\n##################"..., 432) = 432
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -280, SEEK_CUR) = 152
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1bfbcc8e60) = 21725
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x434aa0, [], SA_RESTORER, 0x2b1bfb9b85d0}, {SIG_DFL}, 8) = 0
wait4(-1, glusterfs: FATAL: already running as PID 21665!
[{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 21725
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fffaf540334, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x434aa0, [], SA_RESTORER, 0x2b1bfb9b85d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "/usr/sbin/glusterfsd --pidfile=/"..., 432) = 280
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -187, SEEK_CUR) = 245
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1bfbcc8e60) = 21726
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x434aa0, [], SA_RESTORER, 0x2b1bfb9b85d0}, {SIG_DFL}, 8) = 0
wait4(-1, glusterfs: WARNING: ignoring stale pidfile for PID 21669
[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 21726
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fffaf540334, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x434aa0, [], SA_RESTORER, 0x2b1bfb9b85d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "/usr/sbin/glusterfsd --pidfile=/"..., 432) = 187
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -94, SEEK_CUR) = 338
clone(glusterfs: WARNING: ignoring stale pidfile for PID 21672
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1bfbcc8e60) = 21729
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 21729
wait4(-1, 0x7fffaf540324, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x434aa0, [], SA_RESTORER, 0x2b1bfb9b85d0}, {SIG_DFL}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x434aa0, [], SA_RESTORER, 0x2b1bfb9b85d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "/usr/sbin/glusterfsd --pidfile=/"..., 432) = 94
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -1, SEEK_CUR) = 431
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1bfbcc8e60) = 21732
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x434aa0, [], SA_RESTORER, 0x2b1bfb9b85d0}, {SIG_DFL}, 8) = 0
wait4(-1, glusterfs: WARNING: ignoring stale pidfile for PID 21675
[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 21732
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fffaf540334, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x434aa0, [], SA_RESTORER, 0x2b1bfb9b85d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "\n", 432) = 1
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "", 432) = 0
exit_group(0) = ?

Any more ideas ?

Thanks in advance, Christoph Mallau

SysEleven GmbH
10178 Berlin

christoph mallau <dx0r>
Fri 28 Mar 2008 04:13:59 PM UTC, comment #1:

Hi,
Can you please give the output of 'glusterfs --version', and also fuse version? If its 1.3.7 or older, I would recommend you to try glusterfs 1.3.8preX releases.

Regards,
Amar

Amar Tumballi <amarts>
Project Member
Fri 28 Mar 2008 10:44:29 AM UTC, original submission:

Hi guys,

I am currently experimenting with running glusterfs on a OpenVZ VPS. The VPS' OS is gentoo with a 2.6 kernel.
Using it as a NFS-like filesystem did work without any problems.
Now, I am trying to configure a cluster/unify volume wich consists of 4 bricks,
all situated on the VPS. But when I try to start more than one fork of glusterfsd to bring the server volumes online
a la:

Christophs_Teestube ~ # glusterfsd -f /etc/glusterfs/clusterVolSRV{0,1,2,3}.vol,

I get a message like

glusterfsd: WARNING: ignoring stale pidfile for PID 18082

I tried to us the '--pidfile' option to manually create the PID files in /var/run, and used it in a hell script.
Allthough the PID files are all created in /var/run, and allthough they do contain different process iDs, the glusterfsd-process wich are beeing started afterwards will recive SIGINT and then the fork dies immediatly.

Christophs_Teestube ~ # ps w -C glusterfsd
PID TTY STAT TIME COMMAND
18150 ? Ssl 0:00 [glusterfsd]

Christophs_Teestube ~ # ls /var/run -al
total 56
drwxr-xr-x 4 root root 4096 Mar 28 12:13 .
drwxr-xr-x 13 root root 4096 Mar 11 17:21 ..
-rw-r--r-- 1 root root 6 Mar 28 10:37 apache2.pid
-rw-r--r-- 1 root root 5 Mar 28 10:17 cron.pid
--w------- 1 root root 6 Mar 28 12:13 glusterfsd0.pid
--w------- 1 root root 6 Mar 28 12:13 glusterfsd1.pid
--w------- 1 root root 6 Mar 28 12:13 glusterfsd2.pid
--w------- 1 root root 6 Mar 28 12:13 glusterfsd3.pid

Christophs_Teestube ~ # cat /var/run/glusterfsd{0,1,2,3}.pid
18150
18154
18157
18160

I tried to hook the failure by putting an strace, but I have no clue what the problem actually could be.

Christophs_Teestube ~ # strace /etc/init.d/start_cluster
execve("/etc/init.d/start_cluster", ["/etc/init.d/start_cluster"], [/* 22 vars */]) = 0
brk(0) = 0x5c1000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2addaa747000
uname({sys="Linux", node="Christophs_Teestube", ...}) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=19511, ...}) = 0
mmap(NULL, 19511, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2addaa748000
close(3) = 0
open("/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\0\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=373768, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2addaa74d000
mmap(NULL, 1423704, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2addaa74e000
mprotect(0x2addaa79b000, 1048576, PROT_NONE) = 0
mmap(0x2addaa89b000, 57344, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4d000) = 0x2addaa89b000
mmap(0x2addaa8a9000, 2392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2addaa8a9000
close(3) = 0
open("/lib/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=14512, ...}) = 0
mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2addaa949000
mprotect(0x2addaa94b000, 2097152, PROT_NONE) = 0
mmap(0x2addaab4b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x2addaab4b000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\334\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1297552, ...}) = 0
mmap(NULL, 3404792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2addaab4d000
mprotect(0x2addaac83000, 2097152, PROT_NONE) = 0
mmap(0x2addaae83000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x136000) = 0x2addaae83000
mmap(0x2addaae88000, 17400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2addaae88000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2addaae8d000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2addaae8e000
arch_prctl(ARCH_SET_FS, 0x2addaae8ddd0) = 0
mprotect(0x2addaae83000, 16384, PROT_READ) = 0
mprotect(0x2addaab4b000, 4096, PROT_READ) = 0
mprotect(0x2addaa947000, 4096, PROT_READ) = 0
munmap(0x2addaa748000, 19511) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/dev/tty", O_RDWR|O_NONBLOCK) = 3
close(3) = 0
brk(0) = 0x5c1000
brk(0x5e2000) = 0x5e2000
getuid() = 0
getgid() = 0
geteuid() = 0
getegid() = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/proc/meminfo", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2addaa748000
read(3, "MemTotal: 2097152 kB\nMemFre"..., 1024) = 771
close(3) = 0
munmap(0x2addaa748000, 4096) = 0
rt_sigaction(SIGCHLD, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGCHLD, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL}, {SIG_DFL}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_DFL}, 8) = 0
uname({sys="Linux", node="Christophs_Teestube", ...}) = 0
stat("/root", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
getpid() = 18148
getppid() = 18147
socket(PF_FILE, SOCK_STREAM, 0) = 3
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0
socket(PF_FILE, SOCK_STREAM, 0) = 3
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0
open("/etc/nsswitch.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=508, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2addaa748000
read(3, "# /etc/nsswitch.conf:\n# $Header:"..., 4096) = 508
read(3, "", 4096) = 0
close(3) = 0
munmap(0x2addaa748000, 4096) = 0
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=19511, ...}) = 0
mmap(NULL, 19511, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2addaa748000
close(3) = 0
open("/lib/libnss_compat.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\23\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=31432, ...}) = 0
mmap(NULL, 2127080, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2addaae8f000
mprotect(0x2addaae96000, 2093056, PROT_NONE) = 0
mmap(0x2addab095000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x2addab095000
close(3) = 0
open("/lib/libnsl.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20A\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=84792, ...}) = 0
mmap(NULL, 2190032, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2addab097000
mprotect(0x2addab0ab000, 2093056, PROT_NONE) = 0
mmap(0x2addab2aa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13000) = 0x2addab2aa000
mmap(0x2addab2ac000, 6864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2addab2ac000
close(3) = 0
mprotect(0x2addab2aa000, 4096, PROT_READ) = 0
mprotect(0x2addab095000, 4096, PROT_READ) = 0
munmap(0x2addaa748000, 19511) = 0
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=19511, ...}) = 0
mmap(NULL, 19511, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2addaa748000
close(3) = 0
open("/lib/libnss_nis.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220 \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=39288, ...}) = 0
mmap(NULL, 2135256, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2addab2ae000
mprotect(0x2addab2b7000, 2093056, PROT_NONE) = 0
mmap(0x2addab4b6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x2addab4b6000
close(3) = 0
open("/lib/libnss_files.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0` \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=43336, ...}) = 0
mmap(NULL, 2139432, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2addab4b8000
mprotect(0x2addab4c2000, 2093056, PROT_NONE) = 0
mmap(0x2addab6c1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x2addab6c1000
close(3) = 0
mprotect(0x2addab6c1000, 4096, PROT_READ) = 0
mprotect(0x2addab4b6000, 4096, PROT_READ) = 0
munmap(0x2addaa748000, 19511) = 0
open("/etc/passwd", O_RDONLY) = 3
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
lseek(3, 0, SEEK_CUR) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=1319, ...}) = 0
mmap(NULL, 1319, PROT_READ, MAP_SHARED, 3, 0) = 0x2addaa748000
lseek(3, 1319, SEEK_SET) = 1319
munmap(0x2addaa748000, 1319) = 0
close(3) = 0
getpgrp() = 18147
rt_sigaction(SIGCHLD, {0x436110, [], SA_RESTORER, 0x2addaab7d5d0}, {SIG_DFL}, 8) = 0
getrlimit(RLIMIT_NPROC, {rlim_cur=441024, rlim_max=441024}) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/etc/init.d/start_cluster", O_RDONLY) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff003795e0) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
read(3, "#!/bin/bash\n#\n##################"..., 80) = 80
lseek(3, 0, SEEK_SET) = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
dup2(3, 255) = 255
close(3) = 0
fcntl(255, F_SETFD, FD_CLOEXEC) = 0
fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(255, {st_mode=S_IFREG|0755, st_size=392, ...}) = 0
lseek(255, 0, SEEK_CUR) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "#!/bin/bash\n#\n##################"..., 392) = 392
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
stat(".", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
stat("/usr/local/sbin/glusterfsd", {st_mode=S_IFREG|0755, st_size=40342, ...}) = 0
stat("/usr/local/sbin/glusterfsd", {st_mode=S_IFREG|0755, st_size=40342, ...}) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -250, SEEK_CUR) = 142
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2addaae8de60) = 18149
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x434aa0, [], SA_RESTORER, 0x2addaab7d5d0}, {SIG_DFL}, 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 18149
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fff00379174, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x434aa0, [], SA_RESTORER, 0x2addaab7d5d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "glusterfsd --pidfile=/var/run/gl"..., 392) = 250
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -167, SEEK_CUR) = 225
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2addaae8de60) = 18152
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x434aa0, [], SA_RESTORER, 0x2addaab7d5d0}, {SIG_DFL}, 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 18152
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fff00379174, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x434aa0, [], SA_RESTORER, 0x2addaab7d5d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "glusterfsd --pidfile=/var/run/gl"..., 392) = 167
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -84, SEEK_CUR) = 308
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2addaae8de60) = 18156
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x434aa0, [], SA_RESTORER, 0x2addaab7d5d0}, {SIG_DFL}, 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 18156
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fff00379174, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x434aa0, [], SA_RESTORER, 0x2addaab7d5d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "glusterfsd --pidfile=/var/run/gl"..., 392) = 84
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -1, SEEK_CUR) = 391
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2addaae8de60) = 18159
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 18159
wait4(-1, 0x7fff00379164, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x434aa0, [], SA_RESTORER, 0x2addaab7d5d0}, {SIG_DFL}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x434aa0, [], SA_RESTORER, 0x2addaab7d5d0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "\n", 392) = 1
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "", 392) = 0
exit_group(0) = ?

I wonder if anyone has a clue ??
Thanks in advance.
Kindly, Chris

Christoph Mallau

SysEleven GmbH
Berlin, GERMANY

christoph mallau <dx0r>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by chettai (Posted a comment)
  • -unavailable- added by amarts (Posted a comment)
  • -unavailable- added by dx0r (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 29 Aug 2008 08:14:47 PM UTCamartsOpen/ClosedOpen=>Closed
    Fri 29 Aug 2008 08:14:46 PM UTCamartsOriginator Name=>Amar

    Back to the top


    Powered by Savane 3.1-cleanup1