helpdavfs2 - Support: sr #107952, how can I limit cache write rate...

 
 

sr #107952: how can I limit cache write rate to avoid XXL cache ?

Submitted by:  None
Submitted on:  Thu 09 Feb 2012 02:39:28 PM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 3 - NormalStatus: In Progress
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Originator Email: -unavailable-Open/Closed: Open
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 07 Mar 2012 08:28:32 PM UTC, comment #10:

From what your report and the debug-log show davfs2 behaves as intended.

Note: all open files will be held in cache. There is no way around this.

Some uploads failed (neon error: could not read response). davfs2 retried the upload. Sometimes the second attempt succeeded. For at least one file the second attempt failed with HTTP error 412. This means the server refused to accept the file because it has been changed on the server in between. When this happens davfs2 will move the file into lost+found. Files in lost+found use cache space and can not be removed from cache by davfs2.

What you can do to reduce the number of open files:
set option 'delay_upload 0'. davfs2 will now try to uplod the file as soon as it is closed blocking the file system interface as long as the upload is in progress. This will slow down the file system but should reduce the number of open files. If your application opens many files in parallel it will not help. If the first attempt to upload fails the file will stay in the cache until the next - delayed - attempt.

Cache size:
Best would be to give your cache enough space according to the amount of data you want to transfere in one go. There is no reason why the cache has to be on /var. With option 'cache_dir' you can put the cache wherever you want.
You should consider to mount as normal user and not as root. For normal users the cache will by default be placed in the users home directory.

davfs2 and rsync:
Some actions of rsync depend on time stamps. When a file is not held in the cache and the file system is umounted and mounted again file times will probably change as it is not possible to keep the server's time stamps and the local time stamps exactly the same (the server has a different set of time stamps than a unix file system). With option 'delay_upload 0' the local mtime and the Last-Modified date on the server should be in sync. But when uploads fail on the first attempt and must be retried some time later this is no longer true (davfs2 can't completely block the file system until the upload succeeds).
So not everthing that rsync can do will work reliably with davfs2.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 07 Mar 2012 09:07:07 AM UTC, comment #9:

just a note about something I didn't see at first look :

As I didn't have time to continue testing, I have interupted my second "cp" (19:42 in debug log) and after that the cache remained at 57M instead of 50

Anonymous
Tue 06 Mar 2012 06:45:16 PM UTC, comment #8:

my tests :

ScarabeeDuo:/mnt/ovh/mp3/Guns N' Roses/Greatest Hits (EU Version)# cat * >/dev/null

=> sometimes the cache grows up to 60M instead of 50M limit, but it is shrinked a little bit later below 50M ==> OK

ScarabeeDuo:/mnt/ovh# cp -r /home/documents/share/mp3/Guns\ N\'\ Roses tests/

=> the cache grows up to 180M ! then slowly goes down until 42M
==> OK but Why does the cache grows so much ? it would be more secure to limit rate (file per file). If I had copied 5G, my cache partition would have been full, which can be harmfull for mysql or other applications wich use /var disk space.

Let's try again with the same command (wich will overwrite existing files)
=> first, the cache stays around 50M, but finally it grows up slowly, up to 90M and then it will slowly decrease as data is being sent to the server
==> OK. Any Idea about those different behaviours ?

Next time I'll try to run the test with rsync and bandwidth limit as I did until now.

(file #25276)

Anonymous
Wed 29 Feb 2012 07:39:39 PM UTC, comment #7:

As long as the cache does not exceed its limit (50 MiByte) there is no need for davfs2 to remove files from the cache (except they are stale). To see what is going wrong we need a case where the cache is bigger than the limit.

You may decrease the cache limit for these tests (option 'cache_size 5' for example).

Before trying more uploads you may first try to open many existing files. davfs2 will put these files into the cache as well. As long as the files are open they will stay in the cache - not caring about the cache size limit. But when you close the files davfs2 should - after some 10 seconds - reduce the cache size.

If this works properly you should again create many big files until the cache overflows and send the logs.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon 27 Feb 2012 10:20:07 PM UTC, comment #6:

So ... I have uploaded a few new photos to the webdav drive, waited for the network trafic fall down to ~0Kb/s

before I umount I can see this on the drive :
ScarabeeDuo:/var/cache/davfs2# ls -l /mnt/ovh/photos/divers/franche\ comte/ornans/*
-rw-rw-rw- 1 root root 530627 févr. 27 2012 /mnt/ovh/photos/divers/franche comte/ornans/ornans11.jpg
-rw-rw-rw- 1 root root 518879 févr. 27 2012 /mnt/ovh/photos/divers/franche comte/ornans/ornans12.jpg
-rwxrwxrwx 1 root root 364720 févr. 12 07:57 /mnt/ovh/photos/divers/franche comte/ornans/ornans 1.jpg
-rwxrwxrwx 1 root root 395171 févr. 12 07:57 /mnt/ovh/photos/divers/franche comte/ornans/ornans 2.jpg
-rwxrwxrwx 1 root root 396762 févr. 12 07:57 /mnt/ovh/photos/divers/franche comte/ornans/ornans 3.jpg
-rwxrwxrwx 1 root root 540345 févr. 12 07:57 /mnt/ovh/photos/divers/franche comte/ornans/ornans 4.jpg
-rwxrwxrwx 1 root root 472562 févr. 12 07:58 /mnt/ovh/photos/divers/franche comte/ornans/ornans 5.jpg
-rw-rw-rw- 1 root root 496609 févr. 27 22:51 /mnt/ovh/photos/divers/franche comte/ornans/ornans 6.jpg
-rwxr-xr-x 1 thierry thierry 546287 janv. 11 2008 /mnt/ovh/photos/divers/franche comte/ornans/ornans 7.jpg
-rw-rw-rw- 1 root root 459722 févr. 27 2012 /mnt/ovh/photos/divers/franche comte/ornans/ornans 8.jpg
-rw-rw-rw- 1 root root 369031 févr. 27 2012 /mnt/ovh/photos/divers/franche comte/ornans/ornans 9.jpg

And this in the cache :
ScarabeeDuo:/var/cache/davfs2# find . -iname '.orna*'
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans12.jpg.DbI1z6-aFdywi
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans 8.jpg.GvMXGa-inLsqs
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans11.jpg.KPOP4E-msjX88
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans 9.jpg.J26uJc-19Zgux
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans 6.jpg.eVyrYz-MJGF8n
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans 7.jpg.bNfkJ9-hQa1OQ

now, umounting filesystem and remounting it ...

Feb 27 23:00:06 ScarabeeDuo mount.davfs: davfs2 1.4.6
Feb 27 23:00:06 ScarabeeDuo mount.davfs: /sbin/mount.davfs https://cloudnas1.ovh.com/cc2979e4ac8083ed45331f3782acbe2c/ /mnt/ovh/ -o rw,dir_mode=777,file_mode=666
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Configuration:
Feb 27 23:00:06 ScarabeeDuo mount.davfs: url: https://cloudnas1.ovh.com/cc2979e4ac8083ed45331f3782acbe2c/
Feb 27 23:00:06 ScarabeeDuo mount.davfs: mount point: /mnt/ovh
Feb 27 23:00:06 ScarabeeDuo mount.davfs: dav_user: davfs2
Feb 27 23:00:06 ScarabeeDuo mount.davfs: dav_group: davfs2
Feb 27 23:00:06 ScarabeeDuo mount.davfs: ignore_home: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: conf: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: user: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: netdev: 1
Feb 27 23:00:06 ScarabeeDuo mount.davfs: mopts: 0xc0ed0006
Feb 27 23:00:06 ScarabeeDuo mount.davfs: add_mopts: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: kernel_fs: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: buf_size: 0 KiB
Feb 27 23:00:06 ScarabeeDuo mount.davfs: uid: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: gid: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: dir_umask: 06000
Feb 27 23:00:06 ScarabeeDuo mount.davfs: file_umask: 06000
Feb 27 23:00:06 ScarabeeDuo mount.davfs: dir_mode: 040777
Feb 27 23:00:06 ScarabeeDuo mount.davfs: file_mode: 0100666
Feb 27 23:00:06 ScarabeeDuo mount.davfs: scheme: https
Feb 27 23:00:06 ScarabeeDuo mount.davfs: host: cloudnas1.ovh.com
Feb 27 23:00:06 ScarabeeDuo mount.davfs: port: 443
Feb 27 23:00:06 ScarabeeDuo mount.davfs: path: /cc2979e4ac8083ed45331f3782acbe2c/
Feb 27 23:00:06 ScarabeeDuo mount.davfs: servercert: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: secrets: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: clicert: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: p_host: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: p_port: 8080
Feb 27 23:00:06 ScarabeeDuo mount.davfs: useproxy: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: askauth: 1
Feb 27 23:00:06 ScarabeeDuo mount.davfs: locks: 1
Feb 27 23:00:06 ScarabeeDuo mount.davfs: lock_owner: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: lock_timeout: 1800 s
Feb 27 23:00:06 ScarabeeDuo mount.davfs: lock_refresh: 60 s
Feb 27 23:00:06 ScarabeeDuo mount.davfs: expect100: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: if_match_bug: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: drop_weak_etags: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: allow_cookie: 1
Feb 27 23:00:06 ScarabeeDuo mount.davfs: precheck: 1
Feb 27 23:00:06 ScarabeeDuo mount.davfs: ignore_dav_header: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: connect_timeout: 10 s
Feb 27 23:00:06 ScarabeeDuo mount.davfs: read_timeout: 10 s
Feb 27 23:00:06 ScarabeeDuo mount.davfs: retry: 10 s
Feb 27 23:00:06 ScarabeeDuo mount.davfs: max_retry: 300 s
Feb 27 23:00:06 ScarabeeDuo mount.davfs: s_charset: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: header: (null)
Feb 27 23:00:06 ScarabeeDuo mount.davfs: sys_cache: /var/cache/davfs2
Feb 27 23:00:06 ScarabeeDuo mount.davfs: cache_dir: /var/cache/davfs2
Feb 27 23:00:06 ScarabeeDuo mount.davfs: backup_dir: lost+found
Feb 27 23:00:06 ScarabeeDuo mount.davfs: cache_size: 50 MiB
Feb 27 23:00:06 ScarabeeDuo mount.davfs: table_size: 1024
Feb 27 23:00:06 ScarabeeDuo mount.davfs: dir_refresh: 60 s
Feb 27 23:00:06 ScarabeeDuo mount.davfs: file_refresh: 1 s
Feb 27 23:00:06 ScarabeeDuo mount.davfs: delay_upload: 10
Feb 27 23:00:06 ScarabeeDuo mount.davfs: gui_optimize: 0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: debug: 0x7
Feb 27 23:00:06 ScarabeeDuo mount.davfs: neon_debug: 0x3
Feb 27 23:00:06 ScarabeeDuo mount.davfs: mountpoint: /mnt/ovh
Feb 27 23:00:06 ScarabeeDuo mount.davfs: mounts in: /proc/mounts
Feb 27 23:00:06 ScarabeeDuo mount.davfs: PID file: /var/run/mount.davfs/mnt-ovh.pid
Feb 27 23:00:06 ScarabeeDuo mount.davfs: changing persona: euid 120, gid 132
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Initializing webdav
Feb 27 23:00:06 ScarabeeDuo mount.davfs: HTTP session to https://cloudnas1.ovh.com:443 begins.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Initializing cache
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Alignment of dav_node: 16
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Checking cache directory
Feb 27 23:00:06 ScarabeeDuo mount.davfs: /var/cache/davfs2/cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: (nil)->0xfce2b0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Reading stored cache data
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfce2b0->0xfe5ce0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe5ce0->0xfe5e00
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe5e00->0xfe5f70
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe5f70->0xfe60a0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe60a0->0xfe61d0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfce2b0->0xfe65d0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe65d0->0xfe6680
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe65d0->0xfe68d0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe65d0->0xfe6a80
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe65d0->0xfe6bc0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe65d0->0xfe6d20
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe65d0->0xfe6e80
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe65d0->0xfe6fe0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfe65d0->0xfe7160
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running pre_send hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request headers:#012OPTIONS /cc2979e4ac8083ed45331f3782acbe2c/ HTTP/1.1#015#012User-Agent: davfs2/1.4.6 neon/0.29.6#015#012Keep-Alive: #015#012Connection: TE, Keep-Alive#015#012TE: trailers#015#012Host: cloudnas1.ovh.com#015#012#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request-line and headers:
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Doing DNS lookup on cloudnas1.ovh.com...
Feb 27 23:00:06 ScarabeeDuo mount.davfs: req: Connecting to 176.31.19.14:443
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Request sent; retry is 0.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [status-line] < HTTP/1.1 200 OK#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Server: nginx/1.0.10-16b14ec-64a911c-8f8989c-89643e8#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [server], Value: [nginx/1.0.10-16b14ec-64a911c-8f8989c-89643e8]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Date: Mon, 27 Feb 2012 22:00:06 GMT#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [date], Value: [Mon, 27 Feb 2012 22:00:06 GMT]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Content-Type: httpd/unix-directory#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [content-type], Value: [httpd/unix-directory]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Connection: keep-alive#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [connection], Value: [keep-alive]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] DAV: 1,2#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [dav], Value: [1,2]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] DAV: <http://apache.org/dav/propset/fs/1>#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [dav], Value: [<http://apache.org/dav/propset/fs/1>]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] MS-Author-Via: DAV#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [ms-author-via], Value: [DAV]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Allow: OPTIONS,GET,HEAD,POST,DELETE,TRACE,PROPFIND,PROPPATCH,COPY,MOVE,LOCK,UNLOCK#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [allow], Value: [OPTIONS,GET,HEAD,POST,DELETE,TRACE,PROPFIND,PROPPATCH,COPY,MOVE,LOCK,UNLOCK]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Content-Length: 0#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [content-length], Value: [0]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] #015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: End of headers.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running post_headers hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running post_send hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Request ends, status 200 class 2xx, error line:#012200 OK
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running destroy hooks.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Request ends.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running pre_send hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request headers:#012PROPFIND /cc2979e4ac8083ed45331f3782acbe2c/ HTTP/1.1#015#012User-Agent: davfs2/1.4.6 neon/0.29.6#015#012Connection: TE#015#012TE: trailers#015#012Host: cloudnas1.ovh.com#015#012Depth: 1#015#012Content-Length: 286#015#012Content-Type: application/xml#015#012#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request-line and headers:
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request body:
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Request sent; retry is 1.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [status-line] < HTTP/1.1 401 Authorization Required#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Server: nginx/1.0.10-16b14ec-64a911c-8f8989c-89643e8#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [server], Value: [nginx/1.0.10-16b14ec-64a911c-8f8989c-89643e8]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Date: Mon, 27 Feb 2012 22:00:06 GMT#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [date], Value: [Mon, 27 Feb 2012 22:00:06 GMT]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Content-Type: text/html; charset=iso-8859-1#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [content-type], Value: [text/html; charset=iso-8859-1]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Connection: keep-alive#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [connection], Value: [keep-alive]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] WWW-Authenticate: Basic realm="Password required"#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [www-authenticate], Value: [Basic realm="Password required"]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Vary: Accept-Encoding#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [vary], Value: [Accept-Encoding]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Content-Length: 401#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [content-length], Value: [401]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] #015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: End of headers.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running post_headers hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Reading 401 bytes of response body.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Got 401 bytes.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running post_send hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running pre_send hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request headers:#012PROPFIND /cc2979e4ac8083ed45331f3782acbe2c/ HTTP/1.1#015#012User-Agent: davfs2/1.4.6 neon/0.29.6#015#012Connection: TE#015#012TE: trailers#015#012Host: cloudnas1.ovh.com#015#012Depth: 1#015#012Content-Length: 286#015#012Content-Type: application/xml#015#012Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx#015#012#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request-line and headers:
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request body:
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Request sent; retry is 1.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [status-line] < HTTP/1.1 207 Multi-Status#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Server: nginx/1.0.10-16b14ec-64a911c-8f8989c-89643e8#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [server], Value: [nginx/1.0.10-16b14ec-64a911c-8f8989c-89643e8]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Date: Mon, 27 Feb 2012 22:00:06 GMT#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [date], Value: [Mon, 27 Feb 2012 22:00:06 GMT]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Content-Type: text/xml; charset="utf-8"#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [content-type], Value: [text/xml; charset="utf-8"]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Connection: keep-alive#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [connection], Value: [keep-alive]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Vary: Accept-Encoding#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [vary], Value: [Accept-Encoding]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Content-Length: 2717#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [content-length], Value: [2717]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] #015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: End of headers.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running post_headers hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Reading 2717 bytes of response body.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Got 2717 bytes.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running post_send hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Request ends, status 207 class 2xx, error line:#012207 Multi-Status
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running destroy hooks.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Request ends.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: updating node: 0xfce2b0->0xfe5ce0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: /cc2979e4ac8083ed45331f3782acbe2c/photos/
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfce2b0->0x1007a20
Feb 27 23:00:06 ScarabeeDuo mount.davfs: added /cc2979e4ac8083ed45331f3782acbe2c/mp3/
Feb 27 23:00:06 ScarabeeDuo mount.davfs: new node: 0xfce2b0->0x1005950
Feb 27 23:00:06 ScarabeeDuo mount.davfs: added /cc2979e4ac8083ed45331f3782acbe2c/images/
Feb 27 23:00:06 ScarabeeDuo mount.davfs: directory updated: (nil)->0xfce2b0
Feb 27 23:00:06 ScarabeeDuo mount.davfs: /cc2979e4ac8083ed45331f3782acbe2c/
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running pre_send hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request headers:#012PROPFIND /cc2979e4ac8083ed45331f3782acbe2c/ HTTP/1.1#015#012User-Agent: davfs2/1.4.6 neon/0.29.6#015#012Connection: TE#015#012TE: trailers#015#012Host: cloudnas1.ovh.com#015#012Depth: 0#015#012Content-Length: 159#015#012Content-Type: application/xml#015#012Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx#015#012#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request-line and headers:
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Sending request body:
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Request sent; retry is 1.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [status-line] < HTTP/1.1 207 Multi-Status#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Server: nginx/1.0.10-16b14ec-64a911c-8f8989c-89643e8#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [server], Value: [nginx/1.0.10-16b14ec-64a911c-8f8989c-89643e8]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Date: Mon, 27 Feb 2012 22:00:06 GMT#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [date], Value: [Mon, 27 Feb 2012 22:00:06 GMT]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Content-Type: text/xml; charset="utf-8"#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [content-type], Value: [text/xml; charset="utf-8"]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Connection: keep-alive#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [connection], Value: [keep-alive]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Vary: Accept-Encoding#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [vary], Value: [Accept-Encoding]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] Content-Length: 340#015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Header Name: [content-length], Value: [340]
Feb 27 23:00:06 ScarabeeDuo mount.davfs: [hdr] #015
Feb 27 23:00:06 ScarabeeDuo mount.davfs: End of headers.
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Running post_headers hooks
Feb 27 23:00:06 ScarabeeDuo mount.davfs: Reading 340 bytes of response body.
Feb 27 23:00:07 ScarabeeDuo mount.davfs: Set signal handler
Feb 27 23:00:07 ScarabeeDuo mount.davfs: Releasing root privileges
Feb 27 23:00:07 ScarabeeDuo mount.davfs: Releasing terminal
Feb 27 23:00:07 ScarabeeDuo mount.davfs: Writing pid file
Feb 27 23:00:07 ScarabeeDuo mount.davfs: Starting message loop
Feb 27 23:00:07 ScarabeeDuo mount.davfs: fuse kernel version 7
Feb 27 23:00:07 ScarabeeDuo mount.davfs: SELECT: 1
Feb 27 23:00:07 ScarabeeDuo mount.davfs: FUSE_INIT:
Feb 27 23:00:07 ScarabeeDuo mount.davfs: version 7.17
Feb 27 23:00:07 ScarabeeDuo mount.davfs: RET: Succès
Feb 27 23:00:18 ScarabeeDuo mount.davfs: SELECT: 0
Feb 27 23:00:18 ScarabeeDuo mount.davfs: tidy: 0 of 17 nodes changed
Feb 27 23:00:18 ScarabeeDuo mount.davfs: cache-size: 5 MiBytes.
Feb 27 23:00:28 ScarabeeDuo mount.davfs: SELECT: 0
Feb 27 23:00:28 ScarabeeDuo mount.davfs: tidy: 0 of 17 nodes changed
Feb 27 23:00:28 ScarabeeDuo mount.davfs: cache-size: 5 MiBytes.
Feb 27 23:00:38 ScarabeeDuo mount.davfs: SELECT: 0
Feb 27 23:00:38 ScarabeeDuo mount.davfs: tidy: 0 of 17 nodes changed
Feb 27 23:00:38 ScarabeeDuo mount.davfs: cache-size: 5 MiBytes.
Feb 27 23:00:48 ScarabeeDuo mount.davfs: SELECT: 0
...
Feb 27 23:15:38 ScarabeeDuo mount.davfs: SELECT: 0
Feb 27 23:15:38 ScarabeeDuo mount.davfs: tidy: 0 of 17 nodes changed
Feb 27 23:15:38 ScarabeeDuo mount.davfs: cache-size: 5 MiBytes.
...

and so on for 20 minutes ...

The index doesn't change :
diff ovh_index_ori cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/index

and cached files are still there (but maybe it's normal : the cache is not full ? ask if you want me to run it until the cache is full) :

ScarabeeDuo:/var/cache/davfs2# find . -iname '.orna*'
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans12.jpg.DbI1z6-aFdywi
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans 8.jpg.GvMXGa-inLsqs
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans11.jpg.KPOP4E-msjX88
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans 9.jpg.J26uJc-19Zgux
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans 6.jpg.eVyrYz-MJGF8n
./cloudnas1.ovh.com-cc2979e4ac8083ed45331f3782acbe2c+mnt-ovh+root/.ornans 7.jpg.bNfkJ9-hQa1OQ

Anonymous
Mon 27 Feb 2012 09:42:04 PM UTC, comment #5:

Sorry for the long time before this answer : I didn't received the notification for your comment :-/

So it's not exactly what you resumed :

  • most of the files are successfully transfered and removed
  • some files (30 - 50 Mo instead of 5 Mo) are WELL transfered but not removed from cache.

I'll try tu run a new test as you explained and post new infos

P.S. : I have a 3.8 MBPS upload bandwidth so that the transfert is quick fast. I'm not sure if there is any limitation on files size, but if there is any, it is over 2GB

Anonymous
Sun 12 Feb 2012 03:47:00 PM UTC, comment #4:

So I misunderstood the situation. Let me try again:

  • most of the files are successfully transfered to the server and removed from the cache
  • some files, preferably big ones, are not transfered to the server, remain in the cache and are not moved to lost+found

This usually happens when an attempt to upload the file fails in a way that indicates that the failure is only temporarily (e.g. because of connection time out). In this case davfs2 will keep the file in the cache and retry uploading from time to time. The retry interval will increase with every failed attempt.

How to test:

If you run in a situation where some big files are not transfered to the server and stay in the cache:

  • unmount the file system (wait until umount returns)
  • create a backup copy of the file index in the cache directory (it might be needed for debugging, but that depends)
  • set option 'debug most'
  • mount the file system
  • do nothing. Make sure no application tries to access the file system.

While doing nothing you may watch the debug log file. What you should see is:

  • in regular intervals (10 seconds) there will be SELECT calls, followed by some information about the cache (size and how many files are changed)
  • if there are files marked as 'changed' davfs2 should do a PUT request to upload them. What now happens is the most important part.

You should wait about 10 to 30 minutes. If things are working normally the number of cached files should go down. If not the debug log will hopefully show what is going wrong.

The debug log file will suffice (syslog contains the same message but additionally messages from other processes).

Cheers
Werner

P.S.: What is the upload bandwidth of your connection to the server? What is the minimum required time to transfer all these files (some GB). Are there any size restrictions set by your provider?

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sun 12 Feb 2012 10:19:38 AM UTC, comment #3:

Hello,

Here are my logs. The "hello world" file is still in the cache after umount. I'd like to add some precisions :
1) not all files remains in the cache :when I upload many files, the cache os cleared correctly excepted for some files
2) It's often the bigest files which remains in the cache and make it overflows it's capacity

I cannot acces server logs : it's a commercial service (http://www.ovh.fr/hubiC/)

Cheers
Thierry

(file #25045, file #25046)

Anonymous
Sun 12 Feb 2012 09:21:50 AM UTC, comment #2:

"(lost+found is empty, and the files in cache are not the last transfered)"
There is something going wrong in the communication between davfs2 and the server. There is no use in trying to transfer huge amount of data. Instead you should use one single small file to debug this:

  • remove the cache
  • set option 'debug most'
  • create one small file ('echo "test" > filename')
  • look at the debug messages in your log files or send them here for me to analyze.

You may additionally search the server's log files.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 10 Feb 2012 07:25:27 AM UTC, comment #1:

I'd like to complete my first report :

Yesterday, after a few hours, I saw that my cache size was still 3GB large. Then I umounted the davfs drive, removed the so big cache (rm -Rf) and re-mounted the drive. Then I ran another rsync but with a bandwidth limitation in order to avoid flooding.

This morning, rsync is finished since a long time (it as transfered 1.3GB) and my cache is 170MB large instead of 50MB. When will it be cleared ? or at least, when will it be truncated to 50MB ? (lost+found is empty, and the files in cache are not the last transfered)

Thanks

Anonymous
Thu 09 Feb 2012 02:39:28 PM UTC, original submission:

Hello,

I'm trying to seng about 1 000 files of ~6MB each one. When I do this, the cache grows up until the /var partition becomes full (4GB later).

Is there any way to limit davfs write rate so tha it will not exceed the max_cache=50 parameter (unless if I send big files wich are larger than the cache )

In my case, the 10 first files (total of 50MB) should be written quickly to the cache, then if I add an 11th file, I would like davfs wait until some cache space is released.

Are there some config parameters I didn't found ? Is there any "easy" way for you to do that ?

Thanks

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #25276:  debug.gz added by None (47kB - application/x-gzip - my debug file)
file #25045:  debug added by None (35kB - application/octet-stream - debug and syslog during my test)
file #25046:  syslog added by None (377kB - application/octet-stream - debug and syslog during my test)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 06 Mar 2012 06:45:16 PM UTCNoneAttached File-=>Added debug.gz, #25276
    Sun 12 Feb 2012 10:19:38 AM UTCNoneAttached File-=>Added debug, #25045
      Attached File-=>Added syslog, #25046
    Sun 12 Feb 2012 09:21:50 AM UTCwbaumannSeverity1 - Wish=>3 - Normal
      StatusNone=>In Progress
      Assigned toNone=>wbaumann

    Back to the top


    Powered by Savane 3.1-cleanup