helpdavfs2 - Support: sr #108158, davfs2 memory consumption...

 
 

sr #108158: davfs2 memory consumption increases over time?

Submitted by:  None
Submitted on:  Fri 12 Oct 2012 06:29:38 PM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 3 - NormalStatus: Done
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Originator Email: -unavailable-Open/Closed: Open
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Fri 13 Jun 2014 12:18:29 PM UTC, comment #30:

Hi Werner,
I have now a running installation not increasing memory usage :-)

My Installation is:

Ubuntu 10.04.2 LTS
DAVFS2 1.5.0 with your Code fix in src/cache.c
Neon 0.29.0 from http://archive.ubuntu.com/ubuntu/pool/main/n/neon27/neon27_0.29.0.orig.tar.gz
compiled with openSSL

Thanks for your last support an have e nice day.
Matthias

Matthias Huber <huber>
Tue 03 Jun 2014 08:31:46 PM UTC, comment #29:

Hello Matthias,

the output of ./configure looks fine and clearly states that it will use the neon library in //usr. That means the headers form the libenon27-dev packages in /usr/include and the library from the libneon27 package in /usr/lib. When it nevertheless tries to use libneon27-gnutls then there is something seriously broken with these packages. Either a bug in the packages or something went wrong with the package management.

Even when you now have a working davfs2 with your locally build neon library you should consider reporting this to ubuntu.

Waiting for a report about not increasing memory usage

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 03 Jun 2014 12:34:19 PM UTC, comment #28:

Hi Werner,

I could now configure, compile and install neon with openSSL on ubuntu.
After that I could configure, compile and install davFS :-)
I will keep an eye onto the memory now.

Cheers Matthias

Matthias Huber <huber>
Tue 03 Jun 2014 09:40:38 AM UTC, comment #27:

Hi Werner,
My answers below:

Did you configure davfs2 before running make and make install?
YES

My output of ./configure:

mahuber@esbtest:~/davFS/davfs2-1.5.0$ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking whether ln -s works... yes
checking whether NLS is requested... yes
checking for msgfmt... /usr/bin/msgfmt
checking for gmsgfmt... /usr/bin/msgfmt
checking for xgettext... /usr/bin/xgettext
checking for msgmerge... /usr/bin/msgmerge
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking for ld used by GCC... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for shared library run path origin... done
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for CFPreferencesCopyAppValue... no
checking for CFLocaleCopyCurrent... no
checking for GNU gettext in libc... yes
checking whether to use NLS... yes
checking where the gettext function comes from... libc
checking for neon-config... /usr/bin/neon-config
checking linking against neon... yes
configure: using neon library 0.29.0
configure: SSL is supported by neon
configure: zlib is supported by neon
configure: IPv6 is supported by neon
configure: LFS is supported by neon
configure: SOCKSv5 is not supported by neon
configure: thread-safe SSL is supported by neon
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking libintl.h usability... yes
checking libintl.h presence... yes
checking for libintl.h... yes
checking langinfo.h usability... yes
checking langinfo.h presence... yes
checking for langinfo.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking mntent.h usability... yes
checking mntent.h presence... yes
checking for mntent.h... yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for stdint.h... (cached) yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking sys/file.h usability... yes
checking sys/file.h presence... yes
checking for sys/file.h... yes
checking sys/mount.h usability... yes
checking sys/mount.h presence... yes
checking for sys/mount.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking syslog.h usability... yes
checking syslog.h presence... yes
checking for syslog.h... yes
checking termios.h usability... yes
checking termios.h presence... yes
checking for termios.h... yes
checking for unistd.h... (cached) yes
checking utime.h usability... yes
checking utime.h presence... yes
checking for utime.h... yes
checking for an ANSI C-conforming const... yes
checking for uid_t in sys/types.h... yes
checking for inline... inline
checking for int16_t... yes
checking for int32_t... yes
checking for int8_t... yes
checking for mode_t... yes
checking for off_t... yes
checking for pid_t... yes
checking for size_t... yes
checking for ssize_t... yes
checking for struct stat.st_blksize... yes
checking whether time.h and sys/time.h may both be included... yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking for uint32_t... yes
checking for uint64_t... yes
checking for working volatile... yes
checking size of void *... 4
checking for unistd.h... (cached) yes
checking for working chown... yes
checking whether closedir returns void... no
checking for error_at_line... yes
checking vfork.h usability... no
checking vfork.h presence... no
checking for vfork.h... no
checking for fork... yes
checking for vfork... yes
checking for working fork... yes
checking for working vfork... (cached) yes
checking type of array argument to getgroups... gid_t
checking for getgroups... yes
checking for working getgroups... yes
checking for library containing getmntent... none required
checking whether gcc needs -traditional... no
checking whether sys/types.h defines makedev... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking types of arguments for select... int,fd_set ,struct timeval
checking for strftime... yes
checking whether lstat correctly handles trailing slash... yes
checking whether stat accepts an empty string... no
checking for utime.h... (cached) yes
checking whether utime accepts a null argument... yes
checking for endpwent... yes
checking for ftruncate... yes
checking for getmntent... (cached) yes
checking for memset... yes
checking for mkdir... yes
checking for nl_langinfo... yes
checking for rpmatch... yes
checking for select... yes
checking for setlocale... yes
checking for strcasecmp... yes
checking for strchr... yes
checking for strdup... yes
checking for strerror... yes
checking for strpbrk... yes
checking for strrchr... yes
checking for strstr... yes
checking for strtol... yes
checking for strtoull... yes
checking for utime... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... 64
configure: creating ./config.status
config.status: creating Makefile
config.status: creating po/Makefile.in
config.status: creating etc/Makefile
config.status: creating man/Makefile
config.status: creating man/de/Makefile
config.status: creating man/es/Makefile
config.status: creating src/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing po-directories commands
config.status: creating po/POTFILES
config.status: creating po/Makefile

Configuration for building davfs2 1.5.0:

Install Prefix: /usr/local
Compiler: gcc
Large File Support: yes
Neon Library: library in /usr (0.29.0)
SSL is supported by neon
National Language Support: libc

Did you purge the davfs2 package from Ubuntu?

YES

Is /sbin/mount.davfs a symbolic
link to your version of mount.davfs which should be in /usr/local/sbin?

YES
As you can see:

ls -l /sbin/mount.davfs*
lrwxrwxrwx 1 root root 27 2014-05-27 11:51 /sbin/mount.davfs -> /usr/local/sbin/mount.davfs

Output of ldd:

mahuber@esbtest:/sbin$ ldd /sbin/mount.davfs
linux-gate.so.1 => (0xb7893000)
libneon-gnutls.so.27 => not found
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb772c000)
/lib/ld-linux.so.2 (0xb7894000)

...davFS seems to use still the libneon-gnutls.so.27..... how can i change this? I think we want to have openssl....

Thanks and cheers
matthias

Matthias Huber <huber>
Wed 28 May 2014 08:21:37 PM UTC, comment #26:

This looks like your version of davfs2 is not compiled against the Neon library of your system.

Did you configure davfs2 before running make and make install?

If not:
Please read file INSTALL in the top level source directory of davfs2.
cd into this directory and run "./configure". When finished the configure script will print the location of the neon library it will use. It should be /usr.
Run "make" and as root run "make install".

If yes:
Look at the output of configure for anything neon related and send it.

Did you purge the davfs2 package from Ubuntu? Is /sbin/mount.davfs a symbolic link to your version of mount.davfs which should be in /usr/local/sbin?

"ldd /sbin/mount.davfs" and "ldd /usr/local/sbin/mount.davfs" will show you all of the used libraries. Both commands should produce the same output.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 27 May 2014 10:10:23 AM UTC, comment #25:

Hi Werner,

I have the following input:
My only 2 installed packages are now:

ii libneon27 0.29.0-1 An HTTP and WebDAV client library
ii libneon27-dev 0.29.0-1 Header and static library files for libneon27

The location of my neon-config is:

/usr/bin/neon-config

I have no other lib in /usr/local

I can install the davfs but when I try to so a mount I receive now:

/sbin/mount.davfs: error while loading shared libraries: libneon-gnutls.so.27: cannot open shared object file: No such file or directory

The outout of
mahuber@esbtest:~/davFS/davfs2-1.5.0$ /usr/bin/neon-config --libs

-Wl,-Bsymbolic-functions -lneon

What's wrong now?

Ragards Matthias

Matthias Huber <huber>
Sat 24 May 2014 05:50:42 PM UTC, comment #24:

I believe I found the bug in Neon, ne_gnutls.c, function ne_ssl_context_destroy.

I've reported this to the Neon developer and you may follow on the Neon mailing list http://lists.manyfish.co.uk/pipermail/neon/2014-May/001567.html

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Thu 22 May 2014 07:19:22 PM UTC, comment #23:

I just looked for Ubuntu packages. They offer the same packages as Debian and most probably they are just the Debian packages.

Your output from neon-config looks like it is not from a Debian package.

Have you another neon library on your system? In /usr/local?
Please try /usr/bin/neon-config --libs.
If you have another neon-library you should use option --with-neon=/usr when configuring davfs2 to use the library from the package.

Werner

P.S. again:
Use this support tracker (https://savannah.nongnu.org/support/?108158) and don't send emails.

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Thu 22 May 2014 06:23:44 PM UTC, comment #22:

I'm confused about Ubuntu. I would have expected they offer the same packages as Debian for this. This would be
libneon27
libneon27-gnutls
libneon27-dev
libneon27-gnutls-dev

If you install all but libneon27-gnutls-dev the configure script of davfs2 would select libneon27 (the OpenSSL version).

neon-config ist part of the -dev package. That you have libneon-27-dev but neon-config shows -lneon-gnutls is strange. Seems that Ubuntu has dropped the OpenSSL version of neon? Please check again with your package manager.

If you really need to build your own neon library you will need
openssl (you should already have this)
libgnutls-dev
the neon sources (take version 3 from the neon website)

In the top-level source directory of neon please run ./configure --help. It will show the option to select the OpenSSL library. You should not change the prefix so it will get installed in /usr/local. You may also need to run ldconfig. Please read the neon documentation for this.

When building davfs2 you will have to call the configure script with option '--with-neon=/usr/local' to select your neon library.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 21 May 2014 06:18:41 PM UTC, comment #21:

Hello Matthias,

today I finished my long running test and I could reproduce the problem.

For both versions of the Neon library (with OpenSSL and with GnuTLS) I did:

  • create 10000 files in 100 directories, each about 15 kB
  • read some hundred of these files
  • delete all the files and directories
  • run the complete cycle up to 7 times in a row

Results:

Neon complied with OpenSSL:
During the first cycle it allocated about 5 MiB of real memory. This is what I expected from the size of 10000 nodes. It never released the memory. But during the next 6 cycless it did not allocate any additional real memory. As far as I know it is quite common not to release memory once allocated. But it could obviously reuse the freed memory in cycle 1 to 6. I think there is no memory leak.

Neon compiled with GnuTLS:
During the first cycle it allocated about 30 MiB of real memory. This is one magnitude more than I expected. In the next cycles it allocated additional memory. After for 4 cycles real memory was up to 100 MiB and I stopped the test. There is a memory leak.

The memory leak is most probably in GnuTLS. I found two bug reports that may refer to the same bug, but I'm not sure.
http://lists.manyfish.co.uk/pipermail/neon/2013-January/001526.html
https://savannah.gnu.org/support/?107804

I will have to investigate this further. It will take some time. Please tell me your versions of libgnutls and libneon (I used libgnutls 2.12.20 and libneon 0.29.6, both from Debian Wheezy).
Maybe there is already a patch somewhere or I can create one.

Workaround:
The easy way to solve this problem at the moment is to use a neon version that is compiled against OpenSSL. Your distribution probably has one and you can have both version of neon installed at the same time.
To compile davfs2 (including the patch for the minor memory leak) you need the -devel package for the OpenSSL version of neon (the GnuTLS-version of the -devel package has to be removed).
neon-config --libs should show which TLS-library is used.

Cheers
Werner

P.S.: Please use my mail-address only when necessary. As much information as possible should be publicly available on this support tracker.

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 14 May 2014 08:01:52 PM UTC, comment #20:

Hello Mathias,

sorry to hear that the problem still exists.

Another check with valgrind showed no further memory leaks. I also did some quick tests with heavy load which showed no problems. The real memory stayed at about 2500 kB.

In the next days I will do some additional tests running over a longer time. I will also test with Neon compiled against OpenSSL (my version uses Neon with GnuTLS).

You may in the mean look whether there is something special with your system that might be related to the problem, e.g. different libraries (libc, xml, tls, neon).

I run a simple Debian Wheezy system. If I can't reproduce the error here, then the memory leak might be in another component involved.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sun 04 May 2014 02:50:08 PM UTC, comment #19:

I found a memory leak (valgrind is my friend).
Every time a file that is not in the cache is downloaded the pointer to the etag-string is replaced without freeing the old string first.

You can easily fix it in the sources (just two lines missing). It is in file src/cache.c, function update_cache_file, around line 2726. The block of code must look like this:

if (!ret) {
if (node->etag) free(node->etag); /* missing free */
node->etag = etag;
if (smtime) {
node->atime = smtime;
node->mtime = smtime;
node->smtime = smtime;
}
node->utime = time(NULL);
set_cache_file_times(node);
cache_size += node->size;
} else {
if (ret == ENOENT) {
delete_cache_file(node->parent);
node->parent->utime = 0;
*flush = 1;
remove_node(node);
}
delete_cache_file(node);
if (etag) free(etag); /* missing free */
}

Regarding option minimize_mem:
This option is only useful if the directory tree is very big. For a rough estimation you may assume about 400 Byte of working memory for every file or directory (depending on the length of file names and etags). If this amount of working memory is not a problem it is better to not use this option.

Please, after running the file system for long enough, tell me whether this patch fixed the problem.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sat 03 May 2014 08:35:51 AM UTC, comment #18:

Thanks for the logs.

There is no error in the last log (file upload works properly9.

I can see two possible reasons for the growing memory usage:

  • occasional upload errors that blow up lost+ found (the less likely reason)
  • a memory leak in davfs2 that is only noticed with long up-times (more likely)

So I will try to find some memory leak. This will take some time because I am no experience in using the tools.

Meanwhile you may do this:

After cleaning lost+found and starting the service again you might in increasing intervalls watch and note:

  • cache size
  • number of files in lost+found
  • memory usage

If any new files appear in lost+found it would be helpful to search the server log for failed upload attempts and the reason.

Out of interest:
top shows about 76 hours of CPU-time. How long has davfs2 been running?

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Thu 01 May 2014 10:20:31 AM UTC, comment #17:

The most obvious problem is that there are about 6000 files in lost+found. davfs2 will put files there when it can't upload new or changed files to the server.

So please do:

- check the real size of your cache directory (it is in /var/cache/mount.davfs/<name fuilt from servername and mout point>.

- look at the files in lost+found and try to find out where they come from.

There seems to bee something wrong with file uploads. To check this do

- set debug option 'debug most'.

- mount the file system.

- try to copy one of the files in lost+found into some other directory of your webdav file system. For the target use a name that is the same as the one in lost+found but without that random string at the end of the file name, beginn with a '-'-character.

- wait about 15 seconds.

- unmount and send me the logs. Please do not include that zillion of messages from snmpd. I'm only intersted in messages from mount.davfs.

After that you should remove all the files in lost+found. If you don't need them you may just remove all files in the cache directory (including the index file). When running the file system again you should regularly check the lost+found directory.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 29 Apr 2014 07:16:53 PM UTC, comment #16:

davfs2 will only remove information from memory if the file or directory has not been accessed for some time (typically about 5 seconds, depending on option file_refresh).

You should first look for any application or user thats scans the file system regularly. Some graphical file managers do that or some daemon that checks for changes in the file system. If there is any you will have to disable it.

If you can't find any such process, please set options 'debug config' and 'debug cache', mount the file system, do some directory listings, wait about 1 miniute and umount the file system.
Search your log files for entries from mount.davfs and send them to me. It may be quite a lot of messages. You may want to send them to my private email address: -unavailable-

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 29 Apr 2014 06:51:31 AM UTC, comment #15:

Hi Werner,
I installed the last version 1.5.0, but we still have
a memory problem....
The RES counter in the "top" for the mount.davfs process is always increasing and inccreaing and after a while the machine begins to swap and kills another important process.
news config parameter minimize_mem is set to 1 but without success

top

1014 davfs2 20 0 1036m 924m 1480 S 0.0 23.0 75:45.02 mount.davfs

mount is:
https://unicard-test.unibe.ch/webdav on /media/unicard-test type davfs (rw,nosuid,nodev,_netdev,uid=1002,gid=0)

Files in mounted folder:

sudo du -sh unicard-test/
57M unicard-test/

Do you have an idea what can i do?

Thanks for your input.
Cheers Matthias

Matthias Huber <huber>
Sun 20 Apr 2014 05:36:23 PM UTC, comment #14:

The patch is now added to the CVS sources branch MAIN and branch rel-1-4-6.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 01 Mar 2013 08:23:52 AM UTC, comment #13:

Like I suspected this periodically cleaning of metadata keeps the performance good at least with our use. Naturally cleaning too often adds some extra work but it's negligible and nothing when compared with the worse case scenarios which are avoided by this. And one can tweak the frequency with file_refresh parameter.

So I'd say go ahead and merge this into the next release!

- Panu

Panu Outinen <poutinen>
Fri 22 Feb 2013 03:07:07 PM UTC, comment #12:

Good news! So far no errors. Ran our collector app couple of times, 20000 files / 1.7GB worth of data each round.

I will test further later on and especially does it help performance-wise us...

- Panu

Panu Outinen <poutinen>
Thu 21 Feb 2013 06:31:13 PM UTC, comment #11:

Sorry,

here is the patch.

Werner

(file #27497)

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 20 Feb 2013 10:05:54 PM UTC, comment #10:

Did you attach the new patch or ?

Panu Outinen <poutinen>
Wed 20 Feb 2013 08:28:30 PM UTC, comment #9:

Hello Panu,

here is another try.

The problem was that a directory may be open for quite some time and the kernel will use inode information after the directory has been closed. It should only do this for 1 second (that's the live time of inode information set by davfs2).

The old patch took only the time when the directory has been refreshed to calculate whether a file node can be deleted. The new patch also takes the time when the directory has been closed (atime) into account.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon 18 Feb 2013 01:36:14 PM UTC, comment #8:

Hmm... it runs longer but still into problems. I'll send the debug log file to you separately.

- Panu

Panu Outinen <poutinen>
Sun 17 Feb 2013 03:11:49 PM UTC, comment #7:

Here is another patch.
I removed option 'minimize_dirs' as it seems to be useless.
It should no longer remove file nodes that belong to an open directory.
Please have another try.

Werner

(file #27478)

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 13 Feb 2013 08:42:02 PM UTC, comment #6:

Hello Panu,

thanks for the log file.

I think I could identify the bug. I'm removing nodes while the directory they belong to is still open for reading.

I will probably come up with a new patch next weekend.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 12 Feb 2013 07:17:50 PM UTC, comment #5:

Please send it by direct email.
-unavailable-

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 12 Feb 2013 03:58:57 PM UTC, comment #4:

Hi Werner!

I patched the source code of version 1.4.7 (even though the name in the diff file implied version 1.4.6). Compiled ok and I tested against a similar setup we are running daily. Like you said 'minimize_dirs 1' can only be used if one never set current directory inside the davfs2 file system. Just using find command of Linux is enough to trigger this problem (find seem to move it's cwd while running, easily seen with lsof).

But I also ran into problems with running with just 'minimize_mem 1'. Most of the time it works as expected (as seen from the debug log). But sometimes it triggers a situation where a file/files cannot be copied. And these files vary in different runs.

I'm using davfs2.conf file with just these set:
ask_auth 0
use_locks 0
gui_optimize 1
minimize_mem 1
debug most

I have a debug log file with one file having this behavior (stopped our script after noticing this).
This debug file is huge 43MB, should I gzip and attach here or email to you directly ?

- Panu

Panu Outinen <poutinen>
Sun 10 Feb 2013 03:30:12 PM UTC, comment #3:

Hello Panu,

when testing my old patch I noticed a major problem: if the working directory is set to be in the davfs2 file system the kernel will cache the inode number of PWD. davfs2 does not know of this and will remove the node which will cause file system errors.

Because of this I changed the patch and you now have two new configuration options:

  • 'minimize_mem 1' will tell davfs2 to remove unused file nodes, but it will not remove directory nodes. Note that local file times may change unexpectetly because the local times stored in the node may differ from the Last-Modified time stored on the server. You should not use this with applications that depend an file times (like backup programs).
  • additionally setting 'minimize_dirs 1' will tell davfs2 to also remove unused directory nodes. You may use this when your file systems has a lot of directories and you are sure that you never cd into the davfs2 file system.

Attached is a patch.

Please report about your experience so I can decide whether to include it in the next release or not.

Cheers
Werner

(file #27439)

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Thu 07 Feb 2013 08:42:42 AM UTC, comment #2:

Hi Werner!

Back in 2008 I notified about the davfs2 cache growing problem and you got it fixed back then. We've been using it ever since (currently version 1.4.7). And it is under heavy use (nightly packaging scripts with 10000's files going through).

But we are just using it as direct read-only file system access of Windows clients to svn repository through combination of davsf2, tmpfs, autofs & samba. The only problem we've encountered is that over time it slows down if automounter is not given the opportunity to cleanup! This doesn't happen with the automatic scripts itself but people leaving just some executables running. The memory consumption is not really issue for us but my educated guess is that cleaning metadata regularly would help this speed issue too.

So I would be willing to give this patch a test run (next week)!

- Panu

Panu Outinen <poutinen>
Sun 14 Oct 2012 08:53:19 AM UTC, comment #1:

This is a design issue of davfs2 which is not well suited for machines with a small amount of memory.

Details:
davfs2 stores meta data of all ever requested resources in a tree of nodes in the working memory. It only removes nodes that get invalid. When davfs2 is running for some time it might finally store meta information about all resources in memory. This might get rather big if the server holds many resources.

Possible solution:
About two years ago, on request of a user, I added code that will regularly scan the node structure and remove nodes that have not been used for some time. Unfortunately I got no response from the requester of this feature and was not able to seriously test it myself. So I removed that code. (See https://savannah.nongnu.org/support/?107437)

If you are interested to test this feature (and report your experience) I could create a patch for you (or a patched source package). If your tests are successful I would include it as an option in the next release.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 12 Oct 2012 06:29:38 PM UTC, original submission:

10 minutes after a reboot I see mount.davfs2 consuming 1.8% of memory.

After ~18hrs uptime I see mount.davfs2 consuming 13.5% of memory.
After ~21hrs uptime I see mount.davfs2 consuming 15% of memory.

Running:
Linux RASPBERRY 3.2.27+ #160 PREEMPT Mon Sep 17 23:18:42 BST 2012 armv6l GNU/Linux

davfs2 1.4.7 <http://savannah.nongnu.org/projects/davfs2>

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #26754:  davfs2.conf added by None (2kB - application/octet-stream - davfs2.conf file)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by huber (Posted a comment)
  • -unavailable- added by poutinen (Posted a comment)
  • -unavailable- added by wbaumann (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 13 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 24 May 2014 05:50:42 PM UTCwbaumannStatusIn Progress=>Done
    Thu 01 May 2014 10:20:42 AM UTCwbaumannOpen/ClosedClosed=>Open
    Tue 29 Apr 2014 07:16:53 PM UTCwbaumannStatusDone=>In Progress
    Tue 22 Apr 2014 07:46:53 PM UTCwbaumannOpen/ClosedOpen=>Closed
    Sun 20 Apr 2014 05:36:23 PM UTCwbaumannStatusIn Progress=>Done
    Thu 21 Feb 2013 06:31:13 PM UTCwbaumannAttached File-=>Added davfs2-1.4.7-min-mem-1.diff, #27497
    Wed 20 Feb 2013 08:23:55 PM UTCwbaumannAttached File#27478=>Removed
    Sun 17 Feb 2013 03:11:49 PM UTCwbaumannAttached File-=>Added davfs2-1.4.7-min-mem.diff, #27478
    Sun 17 Feb 2013 03:05:12 PM UTCwbaumannAttached File#27439=>Removed
    Sun 10 Feb 2013 03:30:12 PM UTCwbaumannAttached File-=>Added davfs2-1.4.6-min-mem.diff, #27439
    Sun 14 Oct 2012 08:53:19 AM UTCwbaumannStatusNone=>In Progress
      Assigned toNone=>wbaumann
    Fri 12 Oct 2012 06:29:38 PM UTCNoneAttached File-=>Added davfs2.conf, #26754

    Back to the top


    Powered by Savane 3.1-cleanup