maindavfs2 - Support: sr #108539, Improve performance on very slow...

 
 

sr #108539: Improve performance on very slow net link

Submitter:  Massimo <massimo_b>
Submitted:  Wed 02 Apr 2014 07:34:42 AM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  1 - Wish Status:  Wont Do
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 30 Nov 2017 08:22:23 AM UTC, comment #5: 

Even on fast links running unison is still slow for a complete run. There are webdav-synchronizers like FOlderSync on Android doing that a lot faster.

How is the status of multithreaded davfs2 as you said that was on the roadmap?

Massimo <massimo_b>
Tue 08 Apr 2014 08:15:12 AM UTC, comment #4: 

On a self hosted server I would prefer unison with remote shell or socket method or course. This web space as part of a bundle is available via webdav only which is a common case for lot of customers. This is even better than no common interface at all like current situation for Google Drive which has not webdav anymore/yet but Google-API only.

So having a webdav space as central sync point the only solution was unison with davfs2 as far as I know. And it works quite well. A run without changes just for checking changes does not load the whole drive (2GB) but only some MB. Doing that within dir_refresh is even faster. Then I have cache_size as large as the whole space to keep it completely cached.

By default unison does check mtime with fastcheck = yes :
http://www.cis.upenn.edu/~bcpierce/unison/download/releases/beta/unison-manual.html#fastcheck
I'm not sure how that method is matching davfs2 implementation.

I have delay_upload 30 because I was not sure if unison does create temporary lock files. I'm not permanently synchronizing with different devices so delayed upload was no issue. Then I'm only synchronizing when I get to a different desktop machine (I'm the single user). Every desktop machine has the space twice locally that is the ~/.davfs2 cache itself and the target ~/Documents.
Mobile devices are working on the davfs filesystem directly, usually read-only.

I could work with every device on davfs2 directly without unison central synchronization. But due to the slow performance and reliability of davfs2 (at least on slow links) and in order to be functional without internet access I preferred the central sync modell.

Another idea was using some VCS like subversion, mercurial etc. using davfs2. But for ~/Documents I don't need histories and this was hard to disable on common VCS.

The only issue beside the poor single thread performance is that unison sometimes don't detect the latest version and tries to fetch old changes from dav over my new local copy. But most times I know exactly which point is the latest, doing a "put-all" or "get-all" only. But this could be rather an issue of the fastcheck of uinson itself or poor mobile Android apps changing davfs files when reading only.

Massimo <massimo_b>
Mon 07 Apr 2014 05:46:24 PM UTC, comment #3: 

After looking for unison: it seems to work like rsync with some additions.

Running rsync or unison over a WebDAV connection makes no sense. rsync requires a TCP connection between the two computers and must be able to run commands on the remote computer. This is not possible over the HTTP protocol.

By trying to do this you will end up with having to download all files. You will also have to store all of them in the cache or to set 'delay_upload 0' to synchronise file times. But even if you can prevent errors this way, you will loose most of the advantages of rsync and unison, especially efficiency and conflict detection.

You can't do everything over HTTP.

Werner

- <_71007>
Mon 07 Apr 2014 08:11:17 AM UTC, comment #2: 

I already had gui_otimize 1, but since this is not about a file manager but "unison file synchronizer" I disabled that again. Then I have a huge cache to keep the whole virtual disc in cache and try to extent the refresh timeouts:

cache_size      8192
table_size      8192
dir_refresh     7200
file_refresh    3600
delay_upload    30

Massimo <massimo_b>
Sat 05 Apr 2014 06:53:39 PM UTC, comment #1: 

davfs2 is still single threaded. When a file transfer is in progress the file system will be blocked.
Implementing multi-threading is on my TODO-list for many years now but I have still no idea when I will be able to do it.

Graphical file managers tend to cause a lot of unnecessary traffic. Some will open every single file just to show some fancy icons and davfs2 will have to download the file. Some will use polling to update the directory listing and davfs will have to permanently fetch the latest information from the server.
Try to configure your file manager to not do things like this or use another file manager. You may additionally try option 'gui_otimize 1' in your davfs2.conf file.

Werner

- <_71007>
Wed 02 Apr 2014 07:34:42 AM UTC, original submission:  

Hello,

I'm using davfs2 together with unison to synchronize several directories with a central webdav server.
I'm not sure if there is any way to improve the performance on very slow net links like 64 kbit/s. A single file transfer is able to use the complete bandwidth as usual. But:

  • Parallel 'ls' on a directory sometimes does not respond for more than 15 minutes.
  • 'cd' has equal issues
  • Most annoying that 'cd /mnt' where /mnt/webdav.foo is mounted via davfs2 does not respond for very long time blocking the access to other mounts for file managers if direct pathes are not known.


The whole time iftop reports full bandwidth in use on that link but I would like davfs2 to share a bit more bandwidth for parallel tasks.

Best regards,
Massimo

Massimo <massimo_b>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by massimo_b (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-15 _71007 Open/ClosedOpen Closed
    2014-04-07 _71007 StatusIn Progress Wont Do
    2014-04-05 _71007 StatusNone In Progress
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code