maindavfs2 - Support: sr #109015, davfs2 cannot deal correctly with...

 
 

sr #109015: davfs2 cannot deal correctly with "no space left on /var/cache fs"

Submitter:  Askar Safin <safinaskar>
Submitted:  Fri 15 Apr 2016 03:58:41 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 16 Apr 2016 09:07:02 PM UTC, comment #10: 


> But did you check whether your file system works at all.

Yes, it transfered smaller files.

You may close the ticket

Askar Safin <safinaskar>
Sat 16 Apr 2016 06:30:20 PM UTC, comment #9: 

I did not read the message correctly. It says
"can't evaluate PID file name;"
This happens when unmount.davfs tries to evaluate the canonical name of the mount point. The mount point given to umount might be a symbolic link or a relative file name (not starting at /). unmount.davfs does not do any write operations for this. But canonicalizing the file name causes access to the davfs2 file system. This may fail because davfs2 will need space in the cache when you read a directory (davfs2 needs to create a file with the content of the directory).
But it unmounted anyway.

"Okey, how to force them to move from cache to server?"
No need to force davfs2. it will do it anyway.
But did you check whether your file system works at all. Are you sure davfs2 can upload files to the server. There are problems with some servers (mostly because of server bugs). You should not run something like rsync before testing with simple file system operations whether it works.
Can't you check the files on the server by some other means?

To get your file system working you have to give davfs2 enough disk space for its cache (you have 1 TB of it). It is easy to change the location of the cache directory with option cache_dir. You can leave the old cache directory as backup and mount with the new cache directory. Check what is in the file system now. That is what is on the server. Test basic file operations with cp, ls etc.
And after about half a minute: check the lost+found directory. Any file in this directory indicates a problem with the connection to the server.

Hard limit:
You can use Quota for this if you need. But davfs2 needs to cache all open files (even those that are opened read only) and some additional space. Otherwise it cant work or you have to limit file operations to what is possible with the available cache space.
As long as davfs2 can get the required disk space it will take it. And can't see why it should fail when it is possible to succeed. No, I won't implement any hard limit.

Werner

- <_71007>
Sat 16 Apr 2016 10:57:29 AM UTC, comment #8: 

I think it would be very nice if davfs2 will support two cache limits: soft and hard. Soft limit is current cache_size limit with current semantics. Hard limit is very hard limit, it should never exceed even if we are writing to opened files. davfs2 should return "no space" error to app when we exceed this limit. By default this second limit should be equal (for example) to [free space on /var/cache fs] minus [1 MB].

Or alternatively, I can just remove soft limit and put hard limit only.

Askar Safin <safinaskar>
Sat 16 Apr 2016 09:22:15 AM UTC, comment #7: 

Today I experienced this problem again. So, let me again describe what happens in such situation.

At first all is OK and copy files to my webdav disk using rsync.

Then, at some moment rsync gives error:
rsync: write failed on "/mnt/1/data/rmd/start-time-is-2015-02-01-22-38-10.mkv": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.1]

Then, df shows me / is 100% full and "ls /mnt/1" (i. e. that webdav disk) shows:
ls: cannot access /mnt/1: Input/output error

"umount /mnt/1" shows:
/sbin/umount.davfs:
  can't evaluate PID file name;
  trying to unmount anyway;
  please wait for mount.davfs to terminate

After "umount /mnt/1" command /mnt/1 seems to be unmounted (i. e. it doesn't appear in "df").

Then I mount fs again. "mount" gives no error messages. Then "ls /mnt/1". It gives I/O error again. Then "umount" and it gives same error about PID file again.

So, it seems that "umount" needs some space for some reason.

I think you should add some code to prevent full overflow. Please, do something like this: if there is 1 MB (or 10 MB or some other value) left on /var/cache fs, then don't write to fs and return "no space" error to the program. Just to make sure that "umount" will work correctly and will not fail.

Also, what to do next in my situation? You said my files are in cache AND on server, right? Okey, how to force them to move from cache to server?
I just cleaned some space on / (not in cache, it other dirs), then I mounted fs (without errors), then "ls" gave me good output, then unmounted fs (without errors), and then mounted fs (without errors). So, is it true that now all my data moved from cache to server and I may remove cache? Also, you said davfs2 removes files from cache to fit the cache to cache_size (default 50 MiB) when shutting down. But I did my full mount-unmount cycle and my cache size is still 2.3 GB. Why?

Askar Safin <safinaskar>
Sat 16 Apr 2016 08:02:50 AM UTC, comment #6: 

Ok, thanks a lot. You may close this ticket

Askar Safin <safinaskar>
Sat 16 Apr 2016 06:28:42 AM UTC, comment #5: 

There are other possible reasons why davfs2 may not be able to write the PID-file. So the prooposed error text might be misleading in other cases.

I instead decided to follow your first proposal: check for sufficient free disk space when the file system is mounted. To do this right requires some more thinking and changes that I can't do at the moment.

I opened a bug report for this to solve the problem in some later release (bug #47714).

Werner

- <_71007>
Fri 15 Apr 2016 10:37:36 PM UTC, comment #4: 

Thanks a lot.
I am sure that one time "umount" gave me "cannot evaluate PID file name, please, wait for mount.davfs process terminating" message (or some other message very close to this), because I see it in my terminal window. So, I still think you should add to this message: "It is possible your cache is full. Your files are not lost, they are on server or in cache"

Askar Safin <safinaskar>
Fri 15 Apr 2016 08:28:07 PM UTC, comment #3: 

davfs2 will delete files from the cache when the amount of cached data is bigger than the configured cache size. It will do this when it is shut down as well as while it is running.

davfs2 will never delete files that have been changed locally and not been successfully uploaded to the server. This also means that it will not delete files that are in the lost+found-directory. It will not delete files that are open.

Your files are either on the server or in the cache or in both places.

Werner

- <_71007>
Fri 15 Apr 2016 07:04:08 PM UTC, comment #2: 

Does davfs2 delete file from cache when it is closed?

Askar Safin <safinaskar>
Fri 15 Apr 2016 04:53:43 PM UTC, comment #1: 

Yes, the problem is that there is no space left in the cache directory. Due to the nature of WebDAV davfs2 has to hold a local copy of each open file in the cache directory.

Error messages:
While davfs2 is running the only error messages possible are the error messages defined by the file system interface. davfs2 can't send error messages of its own through the file system interface. Because of this missing space in the cache directory will give error "no space left on device".

Error messages when mounting.
I find it hard to anticipate any possible error. What I intend to do instead is to improve documentation (README and the mount.davfs2 man page) to make the need for sufficient cache space more clear.

Solution.
You need enough space on the cache to store all the open files there. davfs2 can't remove open files. You can define another cache directory in your davfs2.conf file wich option "cache_dir" (see man davfs2.conf for details).

Error "cannot evaluate PID file name, ..." on unmounting:
You probably got error "can't write pid file ..." when you mounted. This is because your root partition is full.

Werner

- <_71007>
Fri 15 Apr 2016 03:58:41 PM UTC, original submission:  

My / (and therefore /var/cache) fs is relatively small: something like 10 GB. Also I have huge disk mounted to /root/data: 1-2 TB. And I have davfs2 fs mounted to /mnt/1 with 1-2 TB.

I'm trying to move some data from /root/data to /mnt/1 using "rsync --archive --remove-source-files ...". This means that source files (i. e. at /root/data) are removed when written to /mnt/1.

Sometimes my rsync command failes with the message "no space left on device". Then I unmounted /mnt/1 and mounted it again and "df" showed me that there is a lot of free space at /mnt/1, but there is very little or no free space on / (fs where /var/cache resides).

When I unmount /mnt/1 I see something like this: "cannot evaluate PID file name, please, wait for mount.davfs process terminating". When I mount /mnt/1 I see no error message.

Sometimes something even worse happens: "ls /mnt/1" gives error "Input/output error".

I think the reason for such strange behavior is /var/cache or / overflow.

So, please, make davfs2 to deal correctly with /var/cache overflow. If cache overflows davfs2 should clean it a little on-the-fly. If this is not possible, then, at least give more explanatory error messages, which says to user what exactly happened. Also, this message should say to user whenever user files are safe. For example, example error message (while mounting or while unmounting) may say:

"Your /var/cache is full. Unfortunately, davfs2 cannot automatically deal with this situation. Please, manually unmount your fs, then remove cache and then mount fs again. You files are SAFE (!), i. e. they are NOT LOST due to cache overflow"

Also, please, say me did my files lost?

debian gnu/linux. davfs2 debian package 1.5.2-1

Askar Safin <safinaskar>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by safinaskar (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-17 _71007 StatusIn Progress Done
        Open/ClosedOpen Closed
    2016-04-16 _71007 StatusPostponed In Progress
    2016-04-16 _71007 StatusIn Progress Postponed
    2016-04-15 _71007 StatusNone In Progress
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code