maindavfs2 - Support: sr #108588, 90 second delay to receipt of...

 
 

sr #108588: 90 second delay to receipt of first byte

Submitter:  None
Submitted:  Sat 31 May 2014 08:09:24 PM UTC
Votes: 50
 
Category:  None Priority:  5 - Normal
Severity:  4 - Important Status:  Postponed
Privacy:  Public Assigned to:  _71007
Originator Email:  -email is unavailable- Open/Closed:  Closed
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 04 Jun 2014 05:38:21 PM UTC, comment #3: 

"If the FS was to simply deliver the file without caching then the lack of range requests won't matter, would it?"

What do you mean by "simply deliver". davfs2 is a filesystem. Do be accessible like any other file system it must be integrated into the Linux virtual file system. This is mainly done by the fuse kernel module.

What davfs2 does is to read the upcalls from the kernel and respond to them. When your application opens a file for reading, reads the content and then closes the file again, davfs2 will get a series of upcalls like lookup, lookup, getattr, opendir, readdir, lookup, lookup, ..., close dir, getattr, open (finally this file), read xx bytes from Position yy, the same again and again, maybe many thousend times, and finally close.

But many applications may access the file system at the same time. So the reauests that belong to one open-read-close of a file will not come straight in a row, but will be mixed up with upcalls from many other file system operations, in no particular order.

At some time davfs2 will have to download the file, otherwise it can't serve the read upcalls.

In theory it may be possible to not do the download it one task, but to switch between reading the response from the server and listening for kernel upcalls and answering them. But be sure, this will be seriously more complicated than creating additional threads with all the locking overhead. And you will have to take care to not block the download, because you don't empty the buffer (beeing busy with serving other upcalls).

If you think it's that easy, please look at the user space API of the fuse kernel module and tell me how you would do that (no code needed, just the logic of processing). And please remember: while your apllications only need read access davfs2 is a read and write file system (the write is the reason why there is WebDAV at all).

Werner

- <_71007>
Tue 03 Jun 2014 11:09:20 PM UTC, comment #2: 

Unfortunately tuning down the application is both not plausible (have to support old versions) and unpractical due to the file access time delays will make an existing overbooked scheduler run longer on every effort.  Not your problem, but that's what influences our choices.

I understand #1 and #2, but I'm not following how either one of them means that an option to disable caching won't work. If the FS was to simply deliver the file without caching then the lack of range requests won't matter, would it?

Jo Rhett <jorhett>
Tue 03 Jun 2014 08:05:43 PM UTC, comment #1: 

This problem is not caused by caching. There are two reasons for this behaviour:

  • davfs2 does not yet use ranged requests. When you open a file davfs2 will download the complete file (if it is not in the cache) before it responds. Nowadays most servers (not all) allow to only download parts of a file. So davfs2 could on open download only the first kilo- or megabytes and fetch subsequent junks when they are requested by read operations. This would solve your problem though subsequent read requests will be slower. I intend to implement this but at the moment I have no idea when I will have time enough for this.


  • davfs2 is still single threaded. While one download or upload is in progress davfs2 will not be able to serve any other file system calls. I also intend to change this (indeed I want to this for years now). In a first step I will only try to run uploads in a second thread. Again: no idea when I will have time to do it.


At the moment I can't help much:

  • if the same file is requested more than once then giving davfs2 a lot of disk space for its cache will speed up the second, third and so on request significantly.
  • maybe your applications can be configured for a longer timeout.


Cheers
Werner

- <_71007>
Sat 31 May 2014 08:09:24 PM UTC, original submission:  

We are seeing a problem where access of very large (2-6GB) files is delayed for at least 80 seconds for the receipt of the first byte. Various types of access error out before the first byte is received. Transfer rate is fast - very fast - but total time is several minutes.

This seems to be due to the caching of the entire file in /var/cache/davfs2 prior to delivery of the file.

What options do we have to eliminate this delay?

1. Can we disable this caching? I can't find any options to disable it.

2. What would be required to start delivering the file prior to download being complete?

Our use is read-only, we have zero requirement to write back files. I don't believe that local caching provides any advantage in our scenario.

I apologize for opening a support issue, I can't seem to find a mailing list for this kind of query.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jorhett (Posted a comment)
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by jorhett (Voted in favor of this item)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 50 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-26 _71007 Open/ClosedOpen Closed
    2014-06-04 _71007 StatusIn Progress Postponed
    2014-06-03 _71007 StatusNone In Progress
        Assigned toNone _71007
    2014-06-02 jorhett Carbon-Copy- Added jorhett

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code