maindavfs2 - Support: sr #109865, Unmount forces sync errors and...

 
 

sr #109865: Unmount forces sync errors and files end up in lost+found

Submitter:  James James <jamesjames>
Submitted:  Thu 01 Aug 2019 06:00:30 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 04 Sep 2019 04:55:05 PM UTC, comment #4: 

"even without uploading all files, it still takes about an hour for the unmount to finish"
When davfs2 starts to upload a file it will not stop or do anything else before the upload is finished. I don't know the speed of your internet connection, but when upload takes longer than you would expect from the connection, then there is something else wrong.

"So I was, incorrectly, of the impression that this would be after the unmount has finished."
That was my original intention, until I noticed what would happen on system shutdown.

"but what's a reliable method to determine when all files have been copied and I can unmount/disconnect?"
davfs2 has no such possibility. You can us debug mode (option "debug most") for testing, but you can't use this for normal operation. If you have another access to your WebDAV-resources (Browser?) you can check there.

Hint 1:
If you use davfs2 just to copy files from local to the WebDAV-server and vice versa then you might better use some file transfer tool. cadaver is a simple and efficient tool for this with ftp like user interface. Most graphical file managers can access WebDAV resources. davfs2 only makes sense when you want to access your WebDAV-resources with ordinary programs like editors and the like.

Hint 2:
One common reason that slows down davfs2 are applications than do a lot of file operations - unnoticed by the user. Graphical file managers like to do this. You can use option "debug kernel" and watch your log file with "tail -f " to find these applications (parameter pid is shown for some upcalls). But this does not affect the upload time once the upload started.

Werner

- <_71007>
Wed 28 Aug 2019 12:27:01 PM UTC, comment #3: 

Bump

Anonymous
Thu 08 Aug 2019 10:49:55 PM UTC, comment #2: 

Thanks Werner, for your reply :)

> For this reason davfs2 tries to terminate as quickly as possible into a consistent state.


I understand. However, just FYI, (maybe due to my rather slow internet upload speed?) even without uploading all files, it still takes about an hour for the unmount to finish. If this is "as quickly as possible", I don't know. I could be, like I said, just letting you know.

> But when davfs2 gets the impression that an upload will not be possible because the server does not allow it or there are inconsistencies, then it will move the files into lost+found.


It might very well be a server issue. I don't know. That's okay.

> You may use the mv or cp command to copy files from lost+found into the intended destination directory.


I mean, yes, of course - but we're talking about 300 files here. Maybe in multiple subfolders. That's not going to work on the regular. So there is no built-in possibility to retry after files have already been moved to lost+found, right?

> Give davfs2 enough time or copy the files


I can do that, no problem. However, the only reason why I immediately unmounted the share, is that I want to disconnect from the server once the copying has finished. So I was, incorrectly, of the impression that this would be after the unmount has finished. So I will give davfs2 enough time to upload all files from cache to server - but what's a reliable method to determine when all files have been copied and I can unmount/disconnect?

James James <jamesjames>
Wed 07 Aug 2019 06:55:05 PM UTC, comment #1: 

The reason why on unmount davfs2 does not wait until the synchronisation finishes is this: when the operating system shuts down it unmounts all file systems. It only gives applications some short time to terminate. If they don't terminate in time it will kill them and then you will have real data loss. For this reason davfs2 tries to terminate as quickly as possible into a consistent state.

Files that could not be uploaded will be marked and davfs2 will try again, even after an unmount and a new mount. But when davfs2 gets the impression that an upload will not be possible because the server does not allow it or there are inconsistencies, then it will move the files into lost+found.

I don't know why davfs2 considered uploading the files after the remount as impossible. Most probably the server changed some of the file attributes during the unmount. (When a file that does not yet exist is locked than many servers behave strange and not according to the specs.)

If course it makes no sense to copy files into a davfs2 file system when you already know that there is not time enough to upload these files to the server. Give davfs2 enough time or copy the files only when there is time enough. Keeping files in the cache and trying to upload them the next time is only for those cases, where something does not work as expected. And in case something else goes wrong there is the lost+found directory. So no data loss happend.

You may use the mv or cp command to copy files from lost+found into the intended destination directory.

Werner

- <_71007>
Thu 01 Aug 2019 06:00:30 PM UTC, original submission:  

Hey Werner,

I experienced the following issue and was able to reproduce it multiple times like this:

  • Take a big file of approximately 1,5 GB
  • Split it into 5 MB parts
  • Copy all those small files to a remote WebDAV folder
  • Wait until copying to cache finished
  • Immediately unmount


This results in "waiting while mount.davfs (pid XXXXX) synchronizes the cache" followed by a lot of dots and, after some time, a final "OK".

However, once I mount the share again, a lot of those files haven't been written to the server but are now in lost+found. I've read the docs about lost+found and it says everywhere that there must have been an issue on the server side, but I don't think that this is the case here: If I don't immediately unmount but instead wait until the transfer completed (by waiting until a `ls` of a WebDAV folder doesn't block any more) and then unmount (which doesn't take more than a short moment and doesn't produce the dots), no files are lost.

I think, unmounting should do nothing different than waiting for the sync to finish - so why do I lose the files?

(Also, is there a way to finish the sync afterwards? What do I do with the lost files in lost+found to restore them?)

Thanks! :) Cheers, James

James James <jamesjames>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by jamesjames (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-26 _71007 Open/ClosedOpen Closed
    2019-08-07 _71007 StatusNone Done
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code