maindavfs2 - Support: sr #107064, Filesystem operations locked while...

 
 

sr #107064: Filesystem operations locked while uploading

Submitter:  None
Submitted:  Fri 16 Oct 2009 02:41:37 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Postponed
Privacy:  Public Assigned to:  _71007
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 20 Sep 2016 02:57:52 PM UTC, comment #13: 

If you mount the davfs "above" your home directory you can avoid this problem.  The issue seems to be that the file manager you are using is trying to pull a tree for the directory while davfs is (single threadedly) uploading. 

The solution is probably to add multi-threading to allow pulling down data and uploading at the same time, but mounting shares to /mnt will "fix" it.

 Benjamin Daines <benjamindaines>
Thu 26 May 2016 06:16:29 PM UTC, comment #12: 

davfs2 still is single threaded and I have no idea when this will change. Sorry.

But reading of D-state and the need of kill -9 and also this remark in your original post
"all via fstab mounted filesystems and the whole desktop stalls when waiting for davfs2 while it uploads a file"
I think there are other problems involved, including possibly a kernel bug.

Please look at
https://savannah.nongnu.org/support/?109009
for details.

There is now davfs2-1.5.4 which tries to avoid triggering these problems. It might help in your case too.

Werner

- <_71007>
Thu 26 May 2016 09:40:46 AM UTC, comment #11: 

Sorry, forgot to mention I use davfs2-1.5.2.

 Alex Efros <powerman>
Thu 26 May 2016 09:39:11 AM UTC, comment #10: 

Any progress on this? I've just got process reading from webdav mountpoint in D-state and all other operations like `df` or `stat(webdavmountpoint)` are hangs. I had to kill -9 mount.davfs to unblock everything.

 Alex Efros <powerman>
Sun 29 Jul 2012 08:19:49 PM UTC, comment #9: 

Hello Martin,

sorry for the delay. Due to being sick I'm not able to concentrate as necessary.

I think I understand the overall design and agree with it. Though there surely is a lot of space for optimization I would agree to delay this until the basic version is running.

I think I found two problems that should be fixed:

  • function ensure_not_in_transfer_directory seems to only scan the direct childs of the directory, but it should scan the whole subtree


  • while the ensure_not_in_transfer functions wait the bg_thread might remove  the node and the node pointer becomes invalid. So after waiting the node-pointer has to be revalidated (e.g. by using node_exists).


Some other ideas:

  • instead of using sleep(10) I would prefer to wait on a condition variable.


  • in the old version tidy_cache was called from dav_fuse because it depends on the select call. With the bg_thread this dependency is gone and I would prefer to start the bg_thread either from main or from cache.c.


I hope I will be able to do more serious work soon
Werner

- <_71007>
Wed 25 Jul 2012 07:23:22 PM UTC, comment #8: 

Hi Robert,

in principle you are right. But actually I'm not sure if it is possible to stop a transfer, I guess it must be but I don't know.

What I'm sure is that just blocking is for now the simplest solution, apart from returning with an error, and I'd rather do one simple step after another.

It is still possible to implement something more clever later :-)

Bye,

Martin

 Martin Wache <mw55>
Wed 25 Jul 2012 07:12:32 PM UTC, comment #7: 

Hi,

as my C-knowledge is limited, nevertheless I would like to comment on the locking question.

In case there is a request on a file, which is currently uploaded, the transfer could be stopped and started again once the file is closed. This is closer to the behaviour of dropboxd, because given the file is changed, why should you upload the old version?

robert

 Robert <crobe>
Wed 25 Jul 2012 07:09:30 PM UTC, comment #6: 

Hi Werner,

no need to hurry, there are still some improvements possible anyway. I just wanted to hear your general opinion on this.

About coda, it should not be a big problem do adapt coda, I just didn't look at it. But I don't mind to get rid of it, especially if that means it is possible to simplify code.

About the AVM copyright, I'm not entirely sure. I think there are some small parts left, but they could easily be rewritten if you mind. However, if you don't mind just add them, after all they inspired the whole thing and released their code under gpl.

Thanks,

Martin

 Martin Wache <mw55>
Wed 25 Jul 2012 01:28:11 PM UTC, comment #5: 

Hello Martin,

thanks a lot for the work.

I will nedd about two days to understand (I never wrote a multithreated C-program) and to comment.

Concerning coda:
For quite some time I intend to remove coda-support. That would allow to remove some restrictions required by coda. fuse is part of the Linux-kernel for many years now and I don't think anybody is still using coda.

I was never sure what to put in separate threads. I tended to think of all HTTP-requests. With uploading by tidy_cache beeing the main blocker I believe it is a good idea to start with this background threat.

copyright:
Is there any of AVM's code left in your patch that would require to add AVM to the copyright-holders (additional to you)?

Cheers
Werner

- <_71007>
Tue 24 Jul 2012 07:46:56 PM UTC, comment #4: 

Hi,

I started porting the multithreaded file transfer from AVM's patched davfs2 to the CVS. First I wanted to port everything, but now I'm not so sure if AVM's solution is the best.

Now I have created a minimal version, which has just one background thread in which dav_tidy_cache() is called and the dirty files are uploaded.
As in AVM's version there is one lock to guard everything, it is only released during sleeping, and in case of the background thread during transfer of the files.
One problem is what to do if there is any change requested on a file that is currently in transfer, AVM usually returns errors, but I decided to block on those operations. This is closer to the original behaviour of davfs2.

I guess the patch still needs some testing, I think I broke coda (I didn't even try to compile it), but I would like to hear your opinion on it :-)

Bye,
Martin

(file #26252)

 Martin Wache <mw55>
Sat 17 Oct 2009 03:34:16 PM UTC, comment #3: 

Hi,
thanks for reply, at least its not pebkac :)

 Anonymous
Sat 17 Oct 2009 12:05:23 PM UTC, comment #2: 

No, it is not necessary, it is a shortcoming of davfs2.
davfs2 is still single threaded, so every action will block anything else.
Though there is some work based on version 1.3.x by AVM to make davfs2 multithreaded, I have not time enough to port this to the current version. I also have the feeling that making davfs2 multi threaded requires a serious redisign of many parts of davfs2.
I can't make any promises when this problem will be solved.

Cheers
Werner

 - <_71007>
Fri 16 Oct 2009 02:45:09 PM UTC, comment #1: 

Sorry, I forgot the version, I'm using the archlinux package davfs2 1.4.1-1, kernel 2.6.30-ARCH and the fuse kernel fs.

 Anonymous
Fri 16 Oct 2009 02:41:37 PM UTC, original submission:  

Hi,

first davfs2 really rocks.

My problem is, that while davfs2 uploads data to the WebDAV storage, the davfs2 filesystem is completely locked, operations as 'ls' and 'df' don't work and even worse, these operations wait until the current file upload is complete.
I saw this when KDE4 checks ( I don't know exactly what it does, it shows up in strace ) all via fstab mounted filesystems and the whole desktop stalls when waiting for davfs2 while it uploads a file. This makes uploading bigger files with small bandwidth impossible.
I hope this describes the problem.

Is this really neccessary? I'm not familiar with Linux filesystem interna, is there maybe some Option to prevent this?

Thanks,

robert

 Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #26252:  bg_uploads.diff added by mw55 (33KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by benjamindaines (Posted a comment)
  • -email is unavailable- added by powerman (Posted a comment)
  • -email is unavailable- added by crobe (Posted a comment)
  • -email is unavailable- added by mw55 (Updated the item)
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-07-24 mw55 Attached File- Added bg_uploads.diff, #26252
    2009-10-18 _71007 StatusNone Postponed
    2009-10-17 _71007 Assigned toNone _71007

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.13-ec04.
    Corresponding source code