Fri 01 Mar 2013 08:23:52 AM UTC, comment #13:
Like I suspected this periodically cleaning of metadata keeps the performance good at least with our use. Naturally cleaning too often adds some extra work but it's negligible and nothing when compared with the worse case scenarios which are avoided by this. And one can tweak the frequency with file_refresh parameter.
So I'd say go ahead and merge this into the next release!
- Panu
|
Fri 22 Feb 2013 03:07:07 PM UTC, comment #12:
Good news! So far no errors. Ran our collector app couple of times, 20000 files / 1.7GB worth of data each round.
I will test further later on and especially does it help performance-wise us...
- Panu
|
Thu 21 Feb 2013 06:31:13 PM UTC, comment #11:
Sorry,
here is the patch.
Werner
(file #27497)
|
Wed 20 Feb 2013 10:05:54 PM UTC, comment #10:
Did you attach the new patch or ?
|
Wed 20 Feb 2013 08:28:30 PM UTC, comment #9:
Hello Panu,
here is another try.
The problem was that a directory may be open for quite some time and the kernel will use inode information after the directory has been closed. It should only do this for 1 second (that's the live time of inode information set by davfs2).
The old patch took only the time when the directory has been refreshed to calculate whether a file node can be deleted. The new patch also takes the time when the directory has been closed (atime) into account.
Werner
|
Mon 18 Feb 2013 01:36:14 PM UTC, comment #8:
Hmm... it runs longer but still into problems. I'll send the debug log file to you separately.
- Panu
|
Sun 17 Feb 2013 03:11:49 PM UTC, comment #7:
Here is another patch.
I removed option 'minimize_dirs' as it seems to be useless.
It should no longer remove file nodes that belong to an open directory.
Please have another try.
Werner
(file #27478)
|
Wed 13 Feb 2013 08:42:02 PM UTC, comment #6:
Hello Panu,
thanks for the log file.
I think I could identify the bug. I'm removing nodes while the directory they belong to is still open for reading.
I will probably come up with a new patch next weekend.
Cheers
Werner
|
Tue 12 Feb 2013 07:17:50 PM UTC, comment #5:
Please send it by direct email.
-unavailable-
Werner
|
Tue 12 Feb 2013 03:58:57 PM UTC, comment #4:
Hi Werner!
I patched the source code of version 1.4.7 (even though the name in the diff file implied version 1.4.6). Compiled ok and I tested against a similar setup we are running daily. Like you said 'minimize_dirs 1' can only be used if one never set current directory inside the davfs2 file system. Just using find command of Linux is enough to trigger this problem (find seem to move it's cwd while running, easily seen with lsof).
But I also ran into problems with running with just 'minimize_mem 1'. Most of the time it works as expected (as seen from the debug log). But sometimes it triggers a situation where a file/files cannot be copied. And these files vary in different runs.
I'm using davfs2.conf file with just these set:
ask_auth 0
use_locks 0
gui_optimize 1
minimize_mem 1
debug most
I have a debug log file with one file having this behavior (stopped our script after noticing this).
This debug file is huge 43MB, should I gzip and attach here or email to you directly ?
- Panu
|
Sun 10 Feb 2013 03:30:12 PM UTC, comment #3:
Hello Panu,
when testing my old patch I noticed a major problem: if the working directory is set to be in the davfs2 file system the kernel will cache the inode number of PWD. davfs2 does not know of this and will remove the node which will cause file system errors.
Because of this I changed the patch and you now have two new configuration options:
- 'minimize_mem 1' will tell davfs2 to remove unused file nodes, but it will not remove directory nodes. Note that local file times may change unexpectetly because the local times stored in the node may differ from the Last-Modified time stored on the server. You should not use this with applications that depend an file times (like backup programs).
- additionally setting 'minimize_dirs 1' will tell davfs2 to also remove unused directory nodes. You may use this when your file systems has a lot of directories and you are sure that you never cd into the davfs2 file system.
Attached is a patch.
Please report about your experience so I can decide whether to include it in the next release or not.
Cheers
Werner
(file #27439)
|
Thu 07 Feb 2013 08:42:42 AM UTC, comment #2:
Hi Werner!
Back in 2008 I notified about the davfs2 cache growing problem and you got it fixed back then. We've been using it ever since (currently version 1.4.7). And it is under heavy use (nightly packaging scripts with 10000's files going through).
But we are just using it as direct read-only file system access of Windows clients to svn repository through combination of davsf2, tmpfs, autofs & samba. The only problem we've encountered is that over time it slows down if automounter is not given the opportunity to cleanup! This doesn't happen with the automatic scripts itself but people leaving just some executables running. The memory consumption is not really issue for us but my educated guess is that cleaning metadata regularly would help this speed issue too.
So I would be willing to give this patch a test run (next week)!
- Panu
|
Sun 14 Oct 2012 08:53:19 AM UTC, comment #1:
This is a design issue of davfs2 which is not well suited for machines with a small amount of memory.
Details:
davfs2 stores meta data of all ever requested resources in a tree of nodes in the working memory. It only removes nodes that get invalid. When davfs2 is running for some time it might finally store meta information about all resources in memory. This might get rather big if the server holds many resources.
Possible solution:
About two years ago, on request of a user, I added code that will regularly scan the node structure and remove nodes that have not been used for some time. Unfortunately I got no response from the requester of this feature and was not able to seriously test it myself. So I removed that code. (See https://savannah.nongnu.org/support/?107437)
If you are interested to test this feature (and report your experience) I could create a patch for you (or a patched source package). If your tests are successful I would include it as an option in the next release.
Werner
|
Fri 12 Oct 2012 06:29:38 PM UTC, original submission:
10 minutes after a reboot I see mount.davfs2 consuming 1.8% of memory.
After ~18hrs uptime I see mount.davfs2 consuming 13.5% of memory.
After ~21hrs uptime I see mount.davfs2 consuming 15% of memory.
Running:
Linux RASPBERRY 3.2.27+ #160 PREEMPT Mon Sep 17 23:18:42 BST 2012 armv6l GNU/Linux
davfs2 1.4.7 <http://savannah.nongnu.org/projects/davfs2>
|