helpdavfs2 - Support: sr #109130, Cache does not clear data

 
 

sr #109130: Cache does not clear data

Submitted by:  Petur <petur>
Submitted on:  Fri 02 Sep 2016 08:43:59 AM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 5 - BlockerStatus: Done
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Open/Closed: OpenOperating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 29 May 2017 08:19:16 PM UTC, comment #4:

I didn't thank you, Werner, for your explanation (sorry).

I fixed my issue by slowing down the transfer.

Petur <petur>
Fri 25 Nov 2016 07:08:46 PM UTC, comment #3:

The cache of davfs2 is permanent, i.e. davfs2 will hold as many files in the cache as allowed by the configured cache size (option cache_size) so they can be reused even after unmounting and mounting again.

If there are more files in the cache than allowed by cache_size you should check the lost+found directory. Files in this directory usually indicate a problem with the connection to the server.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 25 Nov 2016 07:36:06 AM UTC, comment #2:

I set delay_upload to 0 but still there are about 10 files in cache, although only one process (rsync) is writing files to the mounted dav2fs folder.

I would expect when the close doesn't return until the upload has finished there should be only one file in cache.

Why are there multiple files in the cache folder?

Kind regards Jeroen

Jeroen <jeroenst>
Fri 02 Sep 2016 02:06:17 PM UTC, comment #1:

How caching works when writing to files:

  • Wenn you open a file for writing then all writes will go to the cache. The file will stay in the cache at least as long as the file is open. There is no way around this because HTTP does not allow for partial uploads and davfs2 can't know when the file will be closed.
  • When the file is closed the file can be send to the server. By default davfs2 will wait for delay_upload seconds (default 10 seconds) before uploading to avoid uploading temporary files that will be deleted soon after close.
  • davfs2 will check the cache every delay_upload seconds for files that it can upload and will also delete files from the cache if they have been saved to the server and the cache size is bigger than the configured cache_size (default 50 MiB).

Also not that davfs2 is still single threaded, so cache mainainance does not really happen in the background.

If you write a lot of (big) files it may happen that the cache grows very big. What you can do:

  • I assume you have checked that uploading files really work (no files in lost+found).
  • Reduce delay_upload in your davfs2.conf but keep it at least 1 second. davfs2 will more frequently try to upload files and to clean the cache.
  • If this is not enough then set "delay_upload 0". Now davfs2 will upload the file on close and close will not return untill the upload is finished (in case of network errors this may not be true). But now davfs2 will clean the cache only every 10 seconds. If your cache still gets out of space you will have to slow down writing so that the cache can hold all the data you write within 10 seconds.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 02 Sep 2016 08:43:59 AM UTC, original submission:

I tried to copy large amount of files (~2To) from a ftp server to a webdav server mounted using davfs2.

I'm connected to the ftp server with lftp and launch a mirror command with the mounted davfs repository as destination.

At some point, the copy stops because davfs2 fill /var/cache/davsf2 until my hard drive is full. It seems davfs2 do not clear the cache even if the file are send to the webdav server.

Is there a obvious solution?

Petur <petur>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jeroenst (Posted a comment)
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by petur (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 30 May 2017 06:07:19 PM UTCwbaumannStatusIn Progress=>Done
    Fri 02 Sep 2016 02:06:17 PM UTCwbaumannStatusNone=>In Progress
      Assigned toNone=>wbaumann

    Back to the top


    Powered by Savane 3.1-cleanup1