maindavfs2 - Support: sr #109130, Cache does not clear data

 
 

sr #109130: Cache does not clear data

Submitter:  Petur <petur>
Submitted:  Fri 02 Sep 2016 08:43:59 AM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  5 - Blocker Status:  Done
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 29 May 2017 08:19:16 PM UTC, comment #4: 

I didn't thank you, Werner, for your explanation (sorry).

I fixed my issue by slowing down the transfer.

Petur <petur>
Fri 25 Nov 2016 07:08:46 PM UTC, comment #3: 

The cache of davfs2 is permanent, i.e. davfs2 will hold as many files in the cache as allowed by the configured cache size (option cache_size) so they can be reused even after unmounting and mounting again.

If there are more files in the cache than allowed by cache_size you should check the lost+found directory. Files in this directory usually indicate a problem with the connection to the server.

Werner

- <_71007>
Fri 25 Nov 2016 07:36:06 AM UTC, comment #2: 

I set delay_upload to 0 but still there are about 10 files in cache, although only one process (rsync) is writing files to the mounted dav2fs folder.

I would expect when the close doesn't return until the upload has finished there should be only one file in cache.

Why are there multiple files in the cache folder?

Kind regards Jeroen

Jeroen <jeroenst>
Fri 02 Sep 2016 02:06:17 PM UTC, comment #1: 

How caching works when writing to files:

  • Wenn you open a file for writing then all writes will go to the cache. The file will stay in the cache at least as long as the file is open. There is no way around this because HTTP does not allow for partial uploads and davfs2 can't know when the file will be closed.


  • When the file is closed the file can be send to the server. By default davfs2 will wait for delay_upload seconds (default 10 seconds) before uploading to avoid uploading temporary files that will be deleted soon after close.


  • davfs2 will check the cache every delay_upload seconds for files that it can upload and will also delete files from the cache if they have been saved to the server and the cache size is bigger than the configured cache_size (default 50 MiB).


Also not that davfs2 is still single threaded, so cache mainainance does not really happen in the background.
 
If you write a lot of (big) files it may happen that the cache grows very big. What you can do:

  • I assume you have checked that uploading files really work (no files in lost+found).


  • Reduce delay_upload in your davfs2.conf but keep it at least 1 second. davfs2 will more frequently try to upload files and to clean the cache.


  • If this is not enough then set "delay_upload 0". Now davfs2 will upload the file on close and close will not return untill the upload is finished (in case of network errors this may not be true). But now davfs2 will clean the cache only every 10 seconds. If your cache still gets out of space you will have to slow down writing so that the cache can hold all the data you write within 10 seconds.


Werner

- <_71007>
Fri 02 Sep 2016 08:43:59 AM UTC, original submission:  

I tried to copy large amount of files (~2To) from a ftp server to a webdav server mounted using davfs2.

I'm connected to the ftp server with lftp and launch a mirror command with the mounted davfs repository as destination.

At some point, the copy stops because davfs2 fill /var/cache/davsf2 until my hard drive is full. It seems davfs2 do not clear the cache even if the file are send to the webdav server.

Is there a obvious solution?

Petur <petur>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jeroenst (Posted a comment)
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by petur (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-26 _71007 Open/ClosedOpen Closed
    2017-05-30 _71007 StatusIn Progress Done
    2016-09-02 _71007 StatusNone In Progress
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code