maindavfs2 - Support: sr #109029, dvfs2 is producing critical...

 
 

sr #109029: dvfs2 is producing critical performance problems on the webdav-Server

Submitter:  Pascal <bigtruite>
Submitted:  Tue 03 May 2016 05:43:13 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  4 - Important Status:  Done
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 09 May 2016 07:32:24 PM UTC, comment #4: 

Thank you very much... You helped me a lot to understand davfs2. :-)

Pascal

Pascal <bigtruite>
Mon 09 May 2016 06:41:35 PM UTC, comment #3: 

TCP-Connections
---------------
Once upon a time a new TCP-Connection was established for every HTTP-request and closed as soon as the response finished. Nowadays usually "persistent" connections are used. But also persistent connections are closed after some time of inactivity. So seeing frequently "Connection to .." is no error. It is ok.

Unnecessary Downloads
---------------------
Whenever an application opens a file in a davfs2 file system then davfs2 needs this file in its local cache (to serve subsequent reading and writing).
- if the file is aleady in the cache it will only download the file if it has been changed remotely
- if it is not in the cache it is necessary to download it.

To use its delta algorithm rsync reeds to reed from files in the davfs2 file system. According to the manual option --whole-file disables the delta algorithm. My expectation was that this would stop reading from the files in davfs2, but I did not use rsync myself.
I did some short testing and noticed that for files that changed locally rsync will always open the backup file in the davfs2 file system, independent of the option --whole-file. So this option will not help.

The only thing you can do to avoid unnecessary downloads is to give davfs2 as much cache memory on your local hard disk as possible (using option cache_size).

debug-log
---------
There is nothing you need to check in the logs and you probably want to turn off debugging. But of course you may use it to see what is going on within davfs2 if you are interested (upcalls from kernel, actions from the davfs2 cache, HTTP-requests). If you need some explanation please ask.

Werner

- <_71007>
Wed 04 May 2016 10:00:05 PM UTC, comment #2: 

Sessions:
---------
So what i did. I added in the /etc/davfs2/davfs2.conf the lignes:

n_cookies 1
debug most

Till now there are no performance problems anymore. :-))))) It seems that it was really a problem of cookies that weren't send. Thank you very much Werner... :-))))

As i am logging with the debug option, i noticed in the /var/log/syslog, that the expression „Connecting to <IP-Adress>“ appeared 7 times. I think that should be ok. What do you mean?

Are there further expressions i should pay attention in the syslog concerning the high number of different sessions in the past?

Certificate problem
-------------------
I will check it out and try later. The „j“ is working good for me now. ;-)

rsync and davfs2
----------------
By using the rsync-option „--whole-file“ the file would be downloaded completley. The content of the webdav-server is about 1,5 GB. So i think i will stay it like it is, because rsync is working good. :-)

Pascal <bigtruite>
Wed 04 May 2016 07:06:35 AM UTC, comment #1: 

Sessions
--------
Most probably the server creates a session cookie and expects the client to include this in every request. By default davfs2 will not send any cookies and so the server will create a new "session" for every request.

You can use option "n_cookies 1" in your davfs2.conf file to allow for one session cookie. (I recommend to not remove the comment characters (#) in davfs2.conf, but to append your configuration options at the end of the file. The list of commented out options is there to show all the available options and their default values.)

With this option the number of "sessions" should be reduced to one. If not it might be necessary to increase the number of allowed cookies. But it would be better to use davfs2's debug options to see what is going on before doing this.

HTTP and WebDAV don't need sessions and cookies. Every request from davfs2 contains all the necessary information to process the request. I have no idea what these "sessions" would be good for. Maybe it is just a habit of webserver programmers or the "sessions" are needed to create session statistics. If you can disable these sessions on the server side you should do it.

Certificate problem
-------------------
Please have a look at the davfs2.conf man page, option "trust_server_cert". For self signed certificates this is probably a better solution than passing "j" to the mount command.

rsync and davfs2
----------------
rsync is known for its effective delta algorithm. To do this rsync will connect with ssh to the remote side and start a rsync process there. This allows rsync to do cheap local file access on the remote machine.

davfs2 looks like a local file system to rsync. It can't know of all the web traffic it creates when accessing the davfs2 file system. So some of the file access that rsync does to be efficient may turn out to be very inefficient because davfs2 has to download every file that is read by rsync.
You should turn off the delta algorithm of rsync with option "--whole-file".

If there is any chance to let rsync access the server directly, without davfs2, this would be a better solution in any respect.

rsync option -a
---------------
davfs2 tries to behave as much as possible like a normal file system. But WebDAV is not very well suited for a file system. davfs2 only uses basic WebDAV properties (and the "execute" property defined by Apache). Almost none of the normal file attributes can be stored in WebDAV properties. The only property that matches a file attribute is the LastModifiedDate (mtime). But this time is always set by the server. The client can't change it. Nothing to store owner, group, access bits ...
davfs2 will maintain these attributes locally (to satisfy applications) but can't store them on the server. When the file system is unmounted they are lost (except when the whole file stays in davfs2's cache).

rsync option -a is the same as -rlptgoD. From these options only -r is really useful. If there are files with the execute bit set then option -p is needed too.

Option -i (you use it already) is indeed very useful when running rsync over davfs2.

If you are sure that there is only one client accessing the WebDAV resources at any time you may save some traffic by disabling WebDAV locks (option "use_locks 0" in davfs2.conf).

Werner

- <_71007>
Tue 03 May 2016 05:43:13 PM UTC, original submission:  

Hello,

i am running on my "bananapi" debian jessie with davfs2 Version 1.5.2 on it. I installed a script that mounts a webdav-Server with the command:

   echo "j" | mount -t davfs $webdav_server $local_directory_bananapi

I am using echo "j" because of a certificate problem... not nice, but it works well..

Then i am mounting my owncloud via webdav with the following command on to my bananapi:

   echo "j" | mount -t davfs $owncloud_server_bananapi $local_directory_owncloud

After this i am syncing the to mounted directories with rsinc:

   rsync -avu --progress -i --log-file=/root/bin/rsynclog $local_directory_bananapi $local_directory_owncloud

Everything works... The script ends after a few minutes...

Problem:
--------
Davfs2 is producing for this sync 400 different sessions instead of one. That's a problem for the webdav-Server.

I didn't edit the davfs2.conf. See the attached files for performance problems and the davfs2.conf.

Help:
-----
Can you help me? Why is davfs2 producing 400 sessions?

Pascal <bigtruite>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #37059:  24_hours.jpg added by bigtruite (22KiB - image/jpeg)
file #37060:  davfs2.conf added by bigtruite (2KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by bigtruite (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-26 _71007 Open/ClosedOpen Closed
    2016-05-18 _71007 StatusIn Progress Done
    2016-05-04 _71007 StatusNone In Progress
        Assigned toNone _71007
    2016-05-03 bigtruite Attached File- Added 24_hours.jpg, #37059
        Attached File- Added davfs2.conf, #37060

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code