helpdavfs2 - Support: sr #108588, 90 second delay to receipt of...

 
 

sr #108588: 90 second delay to receipt of first byte

Submitted by:  None
Submitted on:  Sat 31 May 2014 08:09:24 PM UTC  
Votes:  50  
 
Category: NonePriority: 5 - Normal
Severity: 4 - ImportantStatus: Postponed
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Originator Email: -unavailable-Open/Closed: Open
Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 04 Jun 2014 05:38:21 PM UTC, comment #3:

"If the FS was to simply deliver the file without caching then the lack of range requests won't matter, would it?"

What do you mean by "simply deliver". davfs2 is a filesystem. Do be accessible like any other file system it must be integrated into the Linux virtual file system. This is mainly done by the fuse kernel module.

What davfs2 does is to read the upcalls from the kernel and respond to them. When your application opens a file for reading, reads the content and then closes the file again, davfs2 will get a series of upcalls like lookup, lookup, getattr, opendir, readdir, lookup, lookup, ..., close dir, getattr, open (finally this file), read xx bytes from Position yy, the same again and again, maybe many thousend times, and finally close.

But many applications may access the file system at the same time. So the reauests that belong to one open-read-close of a file will not come straight in a row, but will be mixed up with upcalls from many other file system operations, in no particular order.

At some time davfs2 will have to download the file, otherwise it can't serve the read upcalls.

In theory it may be possible to not do the download it one task, but to switch between reading the response from the server and listening for kernel upcalls and answering them. But be sure, this will be seriously more complicated than creating additional threads with all the locking overhead. And you will have to take care to not block the download, because you don't empty the buffer (beeing busy with serving other upcalls).

If you think it's that easy, please look at the user space API of the fuse kernel module and tell me how you would do that (no code needed, just the logic of processing). And please remember: while your apllications only need read access davfs2 is a read and write file system (the write is the reason why there is WebDAV at all).

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 03 Jun 2014 11:09:20 PM UTC, comment #2:

Unfortunately tuning down the application is both not plausible (have to support old versions) and unpractical due to the file access time delays will make an existing overbooked scheduler run longer on every effort. Not your problem, but that's what influences our choices.

I understand #1 and #2, but I'm not following how either one of them means that an option to disable caching won't work. If the FS was to simply deliver the file without caching then the lack of range requests won't matter, would it?

Jo Rhett <jorhett>
Tue 03 Jun 2014 08:05:43 PM UTC, comment #1:

This problem is not caused by caching. There are two reasons for this behaviour:

  • davfs2 does not yet use ranged requests. When you open a file davfs2 will download the complete file (if it is not in the cache) before it responds. Nowadays most servers (not all) allow to only download parts of a file. So davfs2 could on open download only the first kilo- or megabytes and fetch subsequent junks when they are requested by read operations. This would solve your problem though subsequent read requests will be slower. I intend to implement this but at the moment I have no idea when I will have time enough for this.
  • davfs2 is still single threaded. While one download or upload is in progress davfs2 will not be able to serve any other file system calls. I also intend to change this (indeed I want to this for years now). In a first step I will only try to run uploads in a second thread. Again: no idea when I will have time to do it.

At the moment I can't help much:

  • if the same file is requested more than once then giving davfs2 a lot of disk space for its cache will speed up the second, third and so on request significantly.
  • maybe your applications can be configured for a longer timeout.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sat 31 May 2014 08:09:24 PM UTC, original submission:

We are seeing a problem where access of very large (2-6GB) files is delayed for at least 80 seconds for the receipt of the first byte. Various types of access error out before the first byte is received. Transfer rate is fast - very fast - but total time is several minutes.

This seems to be due to the caching of the entire file in /var/cache/davfs2 prior to delivery of the file.

What options do we have to eliminate this delay?

1. Can we disable this caching? I can't find any options to disable it.

2. What would be required to start delivering the file prior to download being complete?

Our use is read-only, we have zero requirement to write back files. I don't believe that local caching provides any advantage in our scenario.

I apologize for opening a support issue, I can't seem to find a mailing list for this kind of query.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jorhett (Posted a comment)
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by jorhett (Voted in favor of this item)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 50 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 04 Jun 2014 05:38:21 PM UTCwbaumannStatusIn Progress=>Postponed
    Tue 03 Jun 2014 08:05:43 PM UTCwbaumannStatusNone=>In Progress
      Assigned toNone=>wbaumann
    Mon 02 Jun 2014 07:13:41 PM UTCjorhettCarbon-Copy-=>Added jorhett

    Back to the top


    Powered by Savane 3.1-cleanup1