helpdavfs2 - Support: sr #107956, no caching when writing and...


sr #107956: no caching when writing and immediately reading 2GB file

Submitted by:  None
Submitted on:  Wed Feb 15 21:12:15 2012  
Category: NonePriority: 5 - Normal
Severity: 3 - NormalStatus: Done
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Originator Email: -unavailable-Open/Closed: Closed
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


(Jump to the original submission Jump to the original submission)

Mon Feb 20 21:22:50 2012, comment #10:

It's a bug in davfs2. For cache sizes bigger than 4 GiByte there is an integer overflow. The file is deleted because davfs2 stores a wrong value for the cache size.

About line 1891 of your log:

x 08:54:17 x mount.davfs: resize cache: 1921 of 1808 MiBytes used.
x 08:54:17 x mount.davfs: 0 of 1808 MiBytes used.

You can fix this bug in file src/cache.c, around line 636:

max_cache_size = args->cache_size * 0x100000;


max_cache_size = (unsigned long long) args->cache_size * 0x100000;

Please tell me if you need a patch file or a compiled binary (but I only can create binaries for 32-bit x86).


Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon Feb 20 08:29:38 2012, comment #9:

"Should I include reading the file in the log, too?"



Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sun Feb 19 23:55:02 2012, comment #8:


>>The log messages in one of your log files will be really big (every single write request is logged) but it will show what is going on. You may send that big log file by email.

Should I include reading the file in the log, too?


Sun Feb 19 20:13:10 2012, comment #7:

This shows that the problem exists but to solve it we had to know what is the cause.

I can't see any other way than analyzing the traffic. Of course this is some trouble with 2GB. Please use davf2's built-in debugging facilities.

  • set option 'debug most'
  • copy the large file into the davfs2 directory

The log messages in one of your log files will be really big (every single write request is logged) but it will show what is going on. You may send that big log file by email.

I can't do it on my system because

  • my provider restricts file size to 500 MByte (a 2 GB file will stay in the cache forever)
  • hidrive is not acceptable for me. It requires java script to just show the first web page and also blocks the Firefox window without java script enabled. I only would visit a website like this when I'm forced to.


Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sat Feb 18 16:17:57 2012, comment #6:

I succesfully reproduced the problem several times. You could register a free account on hidrive (if it is really related to hidrive). Then do the follwing:
# dd if=/dev/zero of=2GB bs=1 count=1 seek=2000MB
1+0 Datens├Ątze ein
1+0 Datens├Ątze aus
ein Byte kopiert, 3,3934e-05 s, 29,5 kB/s
# echo start: `date +'%s'`; cp 2GB /home/joe/strato/users/nubisave/
start: 1329576913
# echo start: `date +'%s'`; /usr/bin/time -f "%e" cp /home/joe/strato/users/nubisave/2GB /tmp/2GB
start: 1329578921

Also right after reading the file, the cache was empty again:
ll /home/joe/.davfs2/cache/ -h
insgesamt 0

You can see from the time used to read the file that it is not read from the cache. I attach the output of the network activity produced by:
# ifstat -z 1/1 | gawk '( !/KB|eth/ ) {print systime()" "$1" "$2; fflush()}' > network_log

The columns are time_from_the_epoche, in [KB/s], out [KB/s].


(file #25080)

Fri Feb 17 19:20:59 2012, comment #5:

davfs2 will use modification times as cache validator only when no ETag is present. But even in this case davfs2 will store the Last-Modified date from the server additionally to the local mtime. It will not compare the server's Last-Modified date with the local mtime but always compare old server-information with new server-information.
This is not the reason for the problem (unless there is a bug in my code).


Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri Feb 17 19:12:38 2012, comment #4:

davfs2 uses the WebDAV-protocol which is an extension of HTTP. The preferred means for cache validation in HTTP are ETags. An ETag is a short string created by the server. This ETag must change every time the file is changed.

Analyzing the network traffic is a good way to see what is going on and what is going wrong. When you change or create a file in your davfs2 file system this is what should happen:

  • When you close the file davfs2 will make the first attempt to upload the file (if option 'delay_upload' is different from 0 there will be - surprise - some delay).
  • davfs2 will send a request to the server that starts with the line "PUT /path/to/file HTTP/1.1". This will be followed by some header-lines. If the file already existed one of these headers should be "If-Match: ETag-string". The headers will be followed by an empty line and the file contents.
  • The server should respond with line "204 No Content". There will be some header-lines too. One of them may be a header "ETag: ETag-string".
  • Most servers will not send the ETag in response to PUT and davfs2 will do another request immediately after. The request line will be "HEAD /path/to/file HTTP/1.1".
  • The server's response should be "200 OK" and should contain an ETag-header.
  • You will then see probably some PROPFIND-requests. davfs2 uses them to get meta-data about the files.
  • Please watch for "PROPFIND /path/to/ HTTP/1.1". The response should be "207 Multi-Status" and contain an XML-body. This XML-body should have an href-tag for file /path/to/file and associated with this a tag <getEtag> containing the ETag-string.
  • When you read the file davfs2 will send a request to the server. This should be "GET /path/to/file HTTP/1.1". If the file is still in the cache there will be a header "If-Not-Match: ETag-string". If this header is missing the file is no longer in davfs2's cache.
  • The server should respond with "304 Not Changed" and not return the file. If the response is "200 OK" it will also contain the file and it should contain a header "ETag: ETAg-string" with a different value of the ETag-string, meaning the file has been changed on the server.


Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri Feb 17 17:57:40 2012, comment #3:

As I have read elsewhere, when you put a file on the WebDAV server, it's modification date on the server becomes the time it has been uploaded (+/- clock deltas). So the file in your cache cannot have the same last modification timestamp as the file on the server.
I think it's the reason why the file is not kept in your cache.


Thu Feb 16 22:23:57 2012, comment #2:

Thank you for your reply and sorry for being too unprecise. What I wanted to say is that the file is not just read from the cache, but first retrieved from the internet. I can see this behavior by monitoring network activity. Since the cache is large enough I expect davfs to know that I just wrote the exact same file. And as I pointed out, this works for smaller files. So the only explanation I could come up is that either davfs treats large files in a special way or as you pointed out that hidrive tells the client that the cached file has expired. But does the latter make any sense? Why should the file expire, even though it was not touched at all after creating it? I will try to write a small file and wait a while until I read it, to check if the cache expiration is time related. Though I wonder why WebDAV does not use checksums instead to see if the file is still the same.


Thu Feb 16 19:05:22 2012, comment #1:

'Why isn't the 2GB file I write to hidrive cached? The file is directly read from the virtual file system right after it has been copied there.'
I don't understand this. To me 'virtual file system' is part of the Linux kernel. All file system calls will access the virtual file system.

Why do you think the file is not cached. There is no code in davfs2 that would allow to access a file in any way without davfs2 creating a file in the cache.

Of course davfs2 will remove files from the cache when the cache exceeds the cache_size (but only files that are not open can be removed).
davfs2 will also remove files from the cache when it gets information from the WebDAV server that the cached version is outdated.


P.S.: Don't assume that I know anything about 'hidrive'. As a single developer I'm not in a position to know all the brands and names that marketing people invent for WebDAV servers.

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed Feb 15 21:12:15 2012, original submission:


Why isn't the 2GB file I write to hidrive cached? The file is directly read from the virtual file system right after it has been copied there.
There is no problem with other file sizes from 1MB up to 1GB.

My configuration:
#grep -Ev "^ *#|^$" /home/joe/.davfs2/davfs2.conf
use_locks 0
cache_size 10000 # MiByte
delay_upload 0




(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files
file #25080:  network_log added by None (111kB - application/octet-stream - network log)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue Apr 22 19:45:54 2014wbaumannOpen/ClosedOpen=>Closed
    Sun Mar 11 15:42:00 2012wbaumannStatusIn Progress=>Done
    Sat Feb 18 16:17:57 2012NoneAttached File-=>Added network_log, #25080
    Fri Feb 17 19:12:38 2012wbaumannStatusNeed Info=>In Progress
    Thu Feb 16 19:05:22 2012wbaumannStatusNone=>Need Info
      Assigned toNone=>wbaumann

    Back to the top

    Powered by Savane 3.1-cleanup1