helpdavfs2 - Support: sr #107933, Caching problem

 
 

sr #107933: Caching problem

Submitted by:  Jack Douglas <jackdouglas>
Submitted on:  Mon 16 Jan 2012 11:18:23 AM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 3 - NormalStatus: Done
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Open/Closed: ClosedOperating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 01 Feb 2012 08:00:40 PM UTC, comment #7:

In a PROPFIND-response the Etag is included in the body (XML). To see the Etag returned on PROPFIND you must additionally add option 'debug httpbody'. It will increase the volume of the log messages significantly.

davfs2 will only use the Etag when the file is already in the cache. So you should read the file twice with one or two seconds delay.

If everthing works as intended:

  • on PROPFIND / an Etag will be included in the body
  • on response to the first GET an Etag-header will be returned with the same value
  • the second GET-request will have an If-Non-Match header with this Etag value
  • the response to this second request should be

either 305 Not Modified, if the file has not been modified
or 200 OK with a different value in the Etag-header.

It may happen that between the two GET-requests davfs2 will send another PROPFIND-request. If this returns a different Etag value davfs2 will know that the file changed and will not use the If-Non-Match header but do an unconditional request. This should return 200 OK (and the new file).

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 01 Feb 2012 04:09:13 PM UTC, comment #6:

I've had a go at using the built-in debugging capabilities - but I'm not sure how to interpret the resulting log file. Am I right that davfs2 never sees the ETag on the propfind?

(file #24952)

Jack Douglas <jackdouglas>
Mon 30 Jan 2012 07:58:59 PM UTC, comment #5:

'If that is really what is happening then I no longer have a problem'
Why 'if'?
Best thing is to look what is happening. You may use davfs2's built-in debugging capabilities or fetch and analyse the HTTP traffic. You will see what Etags are sent and whether davfs2 does conditional Get with the correct Etag.

'absolutely no caching ever'
Set 'cache_size 0' and 'file_refresh 0'.
davfs2 will still cache any file as long as it is open. After closing the file it may take some 10 seconds before it is removed from the cache. You can reduce this time on the expense of CPU-time by reducing 'delay_upload' (but don't make it 0).

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sat 28 Jan 2012 11:06:06 AM UTC, comment #4:

> Is your Apache older than that?


No, I'm on Debian stable which translates to: 2.2.16

> Does your server send the Etag in response to PROPFIND? The spec requires that you return the exactly same Etag in the body of a PROPFIND response as in a GET response


I've managed to work out that the server does send an eTag but unfortunately I've no way of customizing it (according to this post: http://www.gossamer-threads.com/lists/apache/dev/398886)

Also from that thread: "it sees the etag as having
changed and invalidates its cache of the file on every directory
listing" - actually for my use case that is exactly what I want - absolutely no caching ever :-)

If that is really what is happening then I no longer have a problem - I guess the original problem was that the default eTag Apache serves up for the PROPFIND was matching the previous GET even after the contents had changed - because Apache did not know the contents had changed (the .php file doesn't change just the database generated content) - does that sound like a correct diagnosis to you?

Many thanks once again for your input.
Jack

Jack Douglas <jackdouglas>
Fri 20 Jan 2012 08:14:58 PM UTC, comment #3:

Why 'Last-Modified' did not work:
There was a bug in apache/mod_dav concerning 'If_modified' headers. It should be fixed in apache 2.2.8 (you may have a look at https://issues.apache.org/bugzilla/show_bug.cgi?id=38034). Is your Apache older than that?

"I was a little surprised to find I needed to remount the fs after fixing the ETags on the apache side - would you expect that?"
No - not really. But after thinking about it. When davfs2 has no Etag but only Last-Modified it will use Last-Modified which seems not to work with your server. Unmounting removed the non-permanent part of the cache and davfs2 will get the Etag next time. You may need to additinally remove the permanent cache (just remove the appropriate subdirectory in the cache directory.

But davfs2 also does a lot of PROPFIND requests. From the responses davfs2 should get the Etag (without the need to unmount).
Does your server send the Etag in response to PROPFIND? The spec requires that you return the exactly same Etag in the body of a PROPFIND response as in a GET response (well and besides it would be very useful too).

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Thu 19 Jan 2012 07:04:56 PM UTC, comment #2:

Last modified was set to the current time:
header("Last-Modified: " . gmdate("D, d M Y H:i:s") . " GMT");

I added the ETag header to just output an ever-increasing number - the problem went away!

I was a little surprised to find I needed to remount the fs after fixing the ETags on the apache side - would you expect that?

Thanks very much for your kind assistance.

Jack Douglas <jackdouglas>
Mon 16 Jan 2012 07:00:32 PM UTC, comment #1:

Does your server correctly set the Etag header or at least the Last-Modified header?
On open davfs2 will send a conditional GET request, which depends on correct Etag values.

Please add option 'debug most' to your davfs2.conf file. One of your log files will then contain a lot of debug messages from mount.davfs. They will show the upcalls from kernel-fuse as well as the HTTP requests and responses.

You may append these log messages.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon 16 Jan 2012 11:18:23 AM UTC, original submission:

I have apache/mod_dav and dav2fs all on the same host. dav2fs is used to mount what Apache is serving to the local filesystem - a sort of loopback. The purpose is that the content of the files served is dynamic and comes from a postgres database via php. In this case the file is an email footer/disclaimer that changes periodically and is read by the mail program.

This mostly works well, but when I try to cp a file from the dav directory to another part of the filesystem, it cp does not seem to trigger the 'has the file changed' check in dav2fs - it just gets the cached copy. Note the php file has not changed, only the dynamic content because something has been updated in the database.

On the other hand, if I cp again 5 seconds later, the correct file contents is copied. I've ended up with a script like this:

#!/bin/bash
cp /root/pgdav/disclaimer.html /etc/postfix/disclaimer.html
sleep 5
cp /root/pgdav/disclaimer.html /etc/postfix/disclaimer.html

Caching related headers in php are set as follows:

header("Cache-Control: no-store, no-cache, must-revalidate");
header("Cache-Control: post-check=0, pre-check=0", false);
header("Pragma: no-cache");
I've set file_refresh 0 in davfs2.conf.

Is there some other setting I am missing to change this behaviour, or am I hitting a bug?

Many thanks,
Jack

Jack Douglas <jackdouglas>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #24952:  log added by jackdouglas (45kB - application/octet-stream - debug most)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by jackdouglas (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 20 Apr 2014 06:33:07 PM UTCwbaumannStatusIn Progress=>Done
      Open/ClosedOpen=>Closed
    Wed 01 Feb 2012 04:09:13 PM UTCjackdouglasAttached File-=>Added log, #24952
    Mon 16 Jan 2012 07:00:32 PM UTCwbaumannStatusNone=>In Progress
      Assigned toNone=>wbaumann

    Back to the top


    Powered by Savane 3.1-cleanup