maindavfs2 - Support: sr #109009, davfs hangs in D-state

 
 

sr #109009: davfs hangs in D-state

Submitter:  Armin <arminxg>
Submitted:  Tue 05 Apr 2016 12:24:07 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 25 Nov 2016 07:39:10 AM UTC, comment #7: 

I can confirm after uploading almost 6GB of files the speed of uploading is still good. In version 1.5.2 it dropped to almost 0. Also browsing the folders works although it's slow while uploading files at the same time.

Good job Werner!

Jeroen <jeroenst>
Sun 17 Apr 2016 07:51:36 PM UTC, comment #6: 

I removed the useless call of is_mounted and released version 1.5.4

Werner

- <_71007>
Wed 06 Apr 2016 01:45:41 PM UTC, comment #5: 

I just created the bug report

https://bugzilla.kernel.org/show_bug.cgi?id=115951

for the FUSE - /proc/mounts problem.

Armin

Armin <arminxg>
Wed 06 Apr 2016 09:13:16 AM UTC, comment #4: 

Great. I am glad I was able to explain my findings :-)

Yes, I would like to file a bug report for the /proc/mounts problem. As soon as I found the correct mailing list too.
So far I only found the github project for libfuse, but nothing special for fuse code in kernel.
Maybe I just use the linux kernel mailing list....

Armin

Armin <arminxg>
Wed 06 Apr 2016 08:37:57 AM UTC, comment #3: 

Thanks for the additional information. I think I now see the problems.

is_mounted
----------
You are right in that calling it in the main loop is not needed.
It got there because I modeled the main loop for fuse like the main loop for coda (which was first) and was not aware of all of the differences between fuse and coda and the consequences.
When a davfs2 file system is unmounted, coda will not close the connection to the mount.davfs daemon and the select call in the loop will always return 0. So checking with is_mounted was necessary to get aware of the unmounting.
Different from this fuse will close the connection and unmounting can be detected when select returns an error (other than EINTR) and is_mounted is not necessary.

(The keep_on_running flag indicates that mout.davfs got signal SIGTERM. This stops the loop and mount.davfs will clean its state and finally call umount. But this no longer works due to some change in the kernel. See below.)

kernel bug
----------
I too think that blocking reads to /proc/mounts is a kernel bug. It seems to be related to another kernel probllem that I am trying to work around at the moment.
When the file system is unmounted, fuse will send a lookup upcall to the userspace daemon. If the userspace daemon does not respond it will block umount indefinitely and only root can force unmount.

I intend to file a bug report regarding the unmount problem, but still have to find out the appropriate mailing list.

Could you file a bug report concerning the blocking reads to /proc/mounts?

Werner

- <_71007>
Wed 06 Apr 2016 07:39:58 AM UTC, comment #2: 

D-state is a process state which cannot be killed. You can see it in the process list "ps x" in the STAT column. S=sleeping (idle), R=running(active), T=traced (stopped) and D is waiting for an event inside the kernel without the possibility to send signals to it (UNINTERRUPTIBLE). This state can happen in an system call (like read, write) when waiting for an event.

In the situation of the hang, davfs is waiting inside a system call waiting to complete, it cannot be killed.

I have checked everything else, added debug output and traced davfs with strace. The result is always the is_mounted() function.
Network connection has no problems and when I remove is_mounted(), all works perfectly.

It looks like a race condition inside the kernel (not davfs error). When davfs (at least two fuse based mounts) are active I can even check the /proc/mounts hang by doing "cat /proc/mounts".
I think /proc/mounts should never block, but it does. Maybe the fuse is holding a mutex while waiting for user space response and that mutex is also used in /proc/mounts.
Someone should investigate the fuse and /proc/mounts dependencies and fix that, but removing is_mounted() in davfs is a workaround to have a functioning multi-instance davfs system.

Why is davfs doing this is_mounted() check inside the main loop anyway? The keep_on_running flag seems to be enough to check for exit the loop.

Armin <arminxg>
Tue 05 Apr 2016 08:11:58 PM UTC, comment #1: 

Are you sure about your bug fix? Couldn't there be another reason for the changed performance?

is_mounted only reads from /proc/mounts information that is stored in the kernel. These reads will not cause any access to the davfs2 file systems. (Why should it. The information about mounted file systems only change when a file system is mounted or unmounted.) Additionally it only happens when there are no upcalls from the fuse kernel module.

I think you should first investigate why "the transfer slows down". Is the load on your network connection to big? Are there errors in the communication with the server?
Please check the lost+found directories. Any file in this directory indicates a problem with the server.

What is D-state anyway?

Werner

- <_71007>
Tue 05 Apr 2016 12:24:07 PM UTC, original submission:  

I use Ubuntu server 15.10 with davfs2 version 1.5.2 and I can reproduce a hang of davfs in D-state.

It is easy to reproduce when other davfs mounts are involved.
I use 2 or more davfs mounts to different cloud services for backup. At beginning all is fine. Then the transfer slows down and it takes just minutes to a complete hang of all davfs processes and of course the tools that try to write data to these mounts (like tar).

I was able to debug that a little bit.
In dav_fuse.c in function dav_fuse_loop() a return code of 0 from the select() when timeout was reached causes to a call to function is_mounted(). And that produces the problem. is_mounted() itself accesses the /proc filesystem to check the mounts. But if the mounted filesystems are under heady load, that check takes some time. At that point this davfs mount slows down as well because it waits for the is_mounted(). No when another mount is used, it does extactly the same thing. Result is: all mounts are waiting for something and blocking each other.
As soon as I just remove the code in dav_fuse_loop() to call is_mounted() and ignore it, all works perfectly!

Why is is_mounted() here used anyway?
A break out of the keep_on_running loop in case of "not mounted" condition can be done in a different way without accessing files that may block.

Armin <arminxg>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jeroenst (Posted a comment)
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by arminxg (Submitted the item)
  • -email is unavailable- added by arminxg
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-26 _71007 Open/ClosedOpen Closed
    2016-04-17 _71007 StatusIn Progress Done
    2016-04-05 _71007 StatusNone In Progress
        Assigned toNone _71007
    2016-04-05 arminxg Carbon-Copy- Added arminxg

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code