maindavfs2 - Support: sr #108398, Cache growing forever & then...

 
 

sr #108398: Cache growing forever & then hanging

Submitter:  Rich <richstanton>
Submitted:  Tue 24 Sep 2013 01:28:36 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 07 Oct 2013 04:12:54 PM UTC, comment #9: 

I mounted as user, and it still hung - looking at the logs, it's hanging here again:

sess: Closing connection.

I'll try posting over at ubuntu, thanks for the help so far!

Rich <richstanton>
Thu 03 Oct 2013 04:11:00 PM UTC, comment #8: 

Concerning the root question:
What I wanted to say is "*if* you want to act as root, don't use sudo but get full root permissions beforehand".

For this test I suggest to not mount as root but to mount as a normal unpriveleged user with only the permissions of this user. So for mounting and for accessing the davfs2 file system you should act as a normal user. For doing this test you will have to act as root in only two cases: to edit /etc/fstab and to watch the logfiles.

The reason why I suggest mounting as a normal user is: it will avoid some possible problems with permissions when davfs2 changes its persona from root to the unprivileged user davfs2. When mounting as normal user davfs2 will run as normal user all the time, But the file /sbin/mount.davfs must be setuid root. It will use this only for some tasks like mounting.

Werner

- <_71007>
Thu 03 Oct 2013 09:14:19 AM UTC, comment #7: 

I'll do that now, one quick question though - you say at the beginning that I should do everything as root, but then later you say I should mount the webdav share as a user. Which do you want me to do? User or root?

Rich <richstanton>
Thu 03 Oct 2013 09:02:08 AM UTC, comment #6: 

What I could see so far from your logs is this:

  • after you issued the command to create the file davfs2 checks (with HEAD) whether a file like this already exixtx (no) and creates a new file node.
  • davfs2 now tries to lock that filename on the server. To do this the Neon library has to send a LOCK-request. Before doing so it will close the connection to create a new one for this request (because the request is nonidempotent).
  • but when neon tries to close the connection it hangs. The log messages are "req: Closing connection for non-idempotent request." and "sess: Closing connection.". The next message would be "sess: Connection closed.", but this message never appears. This is probably when the file system hung and you waited several minutes before killing the process.


So there seems to be a problem with the neon library and its acces to the network stack.

As far as I know the davfs2 package in Ubuntu is taken from Debian Wheezy. With davfs2 on Debian Wheezy I can't reproduce this problem (everything works). The problem must be either related to the Ubuntu system or something special with your system. It may also be related to permission settings.

At some time it may be necessary to hand over the problem to the Ubuntu developers. But we can do some additional tests first.

To avoid any additional garbage in the log files and to avoid unecessary interference with other problems it is important to

  • not use sudo. Get real root privileges if you mount davfs2 as root. Do this before any davfs2 related actions.
  • don't do any "ls" and don't open any graphical file utility.
  • don't cd into the mounted davfs2 file system (use absolute file names if needed).


First test I propose is to mount as normal unpriveleged user. Please do this:

  • create an mountpoint in your home directory
  • add a line in /etc/fstab with option "users"
  • open an additional terminal to display your log file with command "tail -f" so you can watch the debug messages live (you will need root permissions for this)
  • open another terminal and issue command "netstat -tc" which will show the connection to the server and update its state every second.
  • now mount the file system with "mount <name of mountpoint>"
  • create one small file with "echo"
  • watch and wait and do not issue any other command
  • network problems may have real long timeouts, please wait up to 5 minutes before killing davfs2.


If davfs2 still hangs with log message "sess: Closing connection." I am out of my wits and you should hand over this bug to the Ubuntu developers.

If it succeeds this time (you will see a successful PUT after about 10 or 20 seconds and the file system will not freeze) we will have to do the next test.

Werner

- <_71007>
Mon 30 Sep 2013 09:46:33 AM UTC, comment #5: 

I did as you asked, and will attach the log now. Oddly, when I ran 'sudo echo test > testfile' I got 'permission denied'. I therefore started again & did 'sudo bash', before running 'echo test > testfile'. That then hung for a few minutes, and I ended up using kill -9 to get control back - I couldn't do any file operations on /mnt/ otherwise.

Rich <richstanton>
Wed 25 Sep 2013 05:38:20 PM UTC, comment #4: 

The log files unfortunately don't show much. Reasons:

  • touch does not write any data. Please use something like "echo test > testfile"
  • there are a lot of files in the cache and probably in lost+found. davfs2 probably tried to upload changed files. Therefore the file system seemed to hang and you killed it before something interesting happend.


Please do this:

  • Remove the complete cache directory /var/cache/davfs2/shared.xxx.uk-STANTON+mnt-SDrive+root/test4.txt-6s3NZn. You may instead rename it if there are usefull data in the changed and cached files so you may later restore these changes by hand.
  • mount the file system
  • create one file with "echo test > <mountpoint>/testfile
  • wait twenty seconds
  • unmount the file system
  • send the log messages


Werner

- <_71007>
Wed 25 Sep 2013 09:26:27 AM UTC, comment #3: 

Typo - I ran touch 'test4.txt' not 'text4.txt'.

Rich <richstanton>
Wed 25 Sep 2013 09:22:54 AM UTC, comment #2: 

Thanks for the reply - setting if_match_bug 1 didn't seem to help. I set the debug option (if_match_bug is still set to 1), mounted the filesystem & ran 'touch text4.txt'. That failed (permission denied) because I wasn't root, so I then ran 'sudo touch text4.txt' which caused everything to hang - couldn't even ls into /mnt/.

I then did kill -9 mount.davfs, deleted the PID, remounted and unmounted the directory.

I'll attach the syslog of that now.....

Rich <richstanton>
Tue 24 Sep 2013 07:52:49 PM UTC, comment #1: 

I don't know which older working version of davfs2 you used. Some versions ago davfs2 changed the default value of option "if_match_bug". It is no longer set to 1. If your server has this bug (e.g. old versions of apache/mod_dav still widely used) this will cause uploads to fail and fill the cache with files that could not be uploaded. Setting option "if_match_bug 1" will heolp in this case.

If this does not help, please set option "debug most", do some few file system operations and the me the output of mount.davfs from your og files.

Werner

- <_71007>
Tue 24 Sep 2013 01:28:36 PM UTC, original submission:  

I have a webdav mounted directory on ubuntu. It's worked fine for years. Recently I updated to Ubuntu 12.04 LTS (davfs2 1.4.5), and since then the webdav share hasn't worked. It's mounted in fstab as:
https://shared.xxx.uk/xxx/        /mnt/SDrive     davfs   defaults        0       0

I have my username/password in /etc/davfs2/secrets.

When I mount it, it appears to mount fine, and I can browse it readily. syslog shows:
/sbin/mount.davfs https://shared.xxx.uk/xxx/ /mnt/SDrive/ -o rw

But I very quickly get the following in syslog:
open files exceed max cache size by 534 MiBytes
And at this point any attempts to access the share result in a hanging shell.

I increased the cache size from 50MiB to 500MiB, but it just filled up & started giving the same error. If I try & 'touch' a file on the mounted directory but it just hangs forever. The same directory mounted as webdav in windows 8 works fine, and I can write to it fine from windows. Any suggestions for troubleshooting?

Rich <richstanton>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #29266:  davlog3 added by richstanton (190KiB - application/octet-stream)
file #29207:  davfs.log added by richstanton (354KiB - application/octet-stream - Log from comment #2)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by richstanton (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-04-20 _71007 StatusIn Progress Done
        Open/ClosedOpen Closed
    2013-09-30 richstanton Attached File- Added davlog3, #29266
    2013-09-25 richstanton Attached File- Added davfs.log, #29207
    2013-09-24 _71007 StatusNone In Progress
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code