helpdavfs2 - Support: sr #107956, no caching when writing and...

 
 

sr #107956: no caching when writing and immediately reading 2GB file

Submitted by:  None
Submitted on:  Wed 15 Feb 2012 09:12:15 PM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 3 - NormalStatus: Done
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Originator Email: -unavailable-Open/Closed: Closed
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 20 Feb 2012 09:22:50 PM UTC, comment #10:

It's a bug in davfs2. For cache sizes bigger than 4 GiByte there is an integer overflow. The file is deleted because davfs2 stores a wrong value for the cache size.

About line 1891 of your log:

x 08:54:17 x mount.davfs: resize cache: 1921 of 1808 MiBytes used.
x 08:54:17 x mount.davfs: 0 of 1808 MiBytes used.

You can fix this bug in file src/cache.c, around line 636:
replace

max_cache_size = args->cache_size * 0x100000;

by

max_cache_size = (unsigned long long) args->cache_size * 0x100000;

Please tell me if you need a patch file or a compiled binary (but I only can create binaries for 32-bit x86).

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon 20 Feb 2012 08:29:38 AM UTC, comment #9:

"Should I include reading the file in the log, too?"

Yes.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sun 19 Feb 2012 11:55:02 PM UTC, comment #8:

Hi,

>>The log messages in one of your log files will be really big (every single write request is logged) but it will show what is going on. You may send that big log file by email.


Should I include reading the file in the log, too?

Johannes

Anonymous
Sun 19 Feb 2012 08:13:10 PM UTC, comment #7:

This shows that the problem exists but to solve it we had to know what is the cause.

I can't see any other way than analyzing the traffic. Of course this is some trouble with 2GB. Please use davf2's built-in debugging facilities.

  • set option 'debug most'
  • copy the large file into the davfs2 directory

The log messages in one of your log files will be really big (every single write request is logged) but it will show what is going on. You may send that big log file by email.

I can't do it on my system because

  • my provider restricts file size to 500 MByte (a 2 GB file will stay in the cache forever)
  • hidrive is not acceptable for me. It requires java script to just show the first web page and also blocks the Firefox window without java script enabled. I only would visit a website like this when I'm forced to.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sat 18 Feb 2012 04:17:57 PM UTC, comment #6:

I succesfully reproduced the problem several times. You could register a free account on hidrive https://www.free-hidrive.com/ (if it is really related to hidrive). Then do the follwing:
# dd if=/dev/zero of=2GB bs=1 count=1 seek=2000MB
1+0 Datens├Ątze ein
1+0 Datens├Ątze aus
ein Byte kopiert, 3,3934e-05 s, 29,5 kB/s
# echo start: `date +'%s'`; cp 2GB /home/joe/strato/users/nubisave/
start: 1329576913
# echo start: `date +'%s'`; /usr/bin/time -f "%e" cp /home/joe/strato/users/nubisave/2GB /tmp/2GB
start: 1329578921
1873.44

Also right after reading the file, the cache was empty again:
ll /home/joe/.davfs2/cache/nubisave.webdav.hidrive.strato.com+home-joe-strato+joe/ -h
insgesamt 0

You can see from the time used to read the file that it is not read from the cache. I attach the output of the network activity produced by:
# ifstat -z 1/1 | gawk '( !/KB|eth/ ) {print systime()" "$1" "$2; fflush()}' > network_log

The columns are time_from_the_epoche, in [KB/s], out [KB/s].

Thanks,
Johannes

(file #25080)

Anonymous
Fri 17 Feb 2012 07:20:59 PM UTC, comment #5:

davfs2 will use modification times as cache validator only when no ETag is present. But even in this case davfs2 will store the Last-Modified date from the server additionally to the local mtime. It will not compare the server's Last-Modified date with the local mtime but always compare old server-information with new server-information.
This is not the reason for the problem (unless there is a bug in my code).

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 17 Feb 2012 07:12:38 PM UTC, comment #4:

davfs2 uses the WebDAV-protocol which is an extension of HTTP. The preferred means for cache validation in HTTP are ETags. An ETag is a short string created by the server. This ETag must change every time the file is changed.

Analyzing the network traffic is a good way to see what is going on and what is going wrong. When you change or create a file in your davfs2 file system this is what should happen:

  • When you close the file davfs2 will make the first attempt to upload the file (if option 'delay_upload' is different from 0 there will be - surprise - some delay).
  • davfs2 will send a request to the server that starts with the line "PUT /path/to/file HTTP/1.1". This will be followed by some header-lines. If the file already existed one of these headers should be "If-Match: ETag-string". The headers will be followed by an empty line and the file contents.
  • The server should respond with line "204 No Content". There will be some header-lines too. One of them may be a header "ETag: ETag-string".
  • Most servers will not send the ETag in response to PUT and davfs2 will do another request immediately after. The request line will be "HEAD /path/to/file HTTP/1.1".
  • The server's response should be "200 OK" and should contain an ETag-header.
  • You will then see probably some PROPFIND-requests. davfs2 uses them to get meta-data about the files.
  • Please watch for "PROPFIND /path/to/ HTTP/1.1". The response should be "207 Multi-Status" and contain an XML-body. This XML-body should have an href-tag for file /path/to/file and associated with this a tag <getEtag> containing the ETag-string.
  • When you read the file davfs2 will send a request to the server. This should be "GET /path/to/file HTTP/1.1". If the file is still in the cache there will be a header "If-Not-Match: ETag-string". If this header is missing the file is no longer in davfs2's cache.
  • The server should respond with "304 Not Changed" and not return the file. If the response is "200 OK" it will also contain the file and it should contain a header "ETag: ETAg-string" with a different value of the ETag-string, meaning the file has been changed on the server.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 17 Feb 2012 05:57:40 PM UTC, comment #3:

Hello,
As I have read elsewhere, when you put a file on the WebDAV server, it's modification date on the server becomes the time it has been uploaded (+/- clock deltas). So the file in your cache cannot have the same last modification timestamp as the file on the server.
I think it's the reason why the file is not kept in your cache.

Cheers
Thierry

Anonymous
Thu 16 Feb 2012 10:23:57 PM UTC, comment #2:

Thank you for your reply and sorry for being too unprecise. What I wanted to say is that the file is not just read from the cache, but first retrieved from the internet. I can see this behavior by monitoring network activity. Since the cache is large enough I expect davfs to know that I just wrote the exact same file. And as I pointed out, this works for smaller files. So the only explanation I could come up is that either davfs treats large files in a special way or as you pointed out that hidrive tells the client that the cached file has expired. But does the latter make any sense? Why should the file expire, even though it was not touched at all after creating it? I will try to write a small file and wait a while until I read it, to check if the cache expiration is time related. Though I wonder why WebDAV does not use checksums instead to see if the file is still the same.

Thanks,
Johannes

Anonymous
Thu 16 Feb 2012 07:05:22 PM UTC, comment #1:

'Why isn't the 2GB file I write to hidrive cached? The file is directly read from the virtual file system right after it has been copied there.'
I don't understand this. To me 'virtual file system' is part of the Linux kernel. All file system calls will access the virtual file system.

Why do you think the file is not cached. There is no code in davfs2 that would allow to access a file in any way without davfs2 creating a file in the cache.

Of course davfs2 will remove files from the cache when the cache exceeds the cache_size (but only files that are not open can be removed).
davfs2 will also remove files from the cache when it gets information from the WebDAV server that the cached version is outdated.

Cheers
Werner

P.S.: Don't assume that I know anything about 'hidrive'. As a single developer I'm not in a position to know all the brands and names that marketing people invent for WebDAV servers.

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 15 Feb 2012 09:12:15 PM UTC, original submission:

Hi,

Why isn't the 2GB file I write to hidrive cached? The file is directly read from the virtual file system right after it has been copied there.
There is no problem with other file sizes from 1MB up to 1GB.

My configuration:
#grep -Ev "^ *#|^$" /home/joe/.davfs2/davfs2.conf
use_locks 0
cache_size 10000 # MiByte
delay_upload 0

Thanks!

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #25080:  network_log added by None (111kB - application/octet-stream - network log)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 22 Apr 2014 07:45:54 PM UTCwbaumannOpen/ClosedOpen=>Closed
    Sun 11 Mar 2012 03:42:00 PM UTCwbaumannStatusIn Progress=>Done
    Sat 18 Feb 2012 04:17:57 PM UTCNoneAttached File-=>Added network_log, #25080
    Fri 17 Feb 2012 07:12:38 PM UTCwbaumannStatusNeed Info=>In Progress
    Thu 16 Feb 2012 07:05:22 PM UTCwbaumannStatusNone=>Need Info
      Assigned toNone=>wbaumann

    Back to the top


    Powered by Savane 3.1-cleanup