maindavfs2 - Support: sr #108697, Corruption on upload during...

 
 

sr #108697: Corruption on upload during extended outage

Submitter:  None
Submitted:  Sat 06 Dec 2014 01:13:12 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  4 - Important Status:  In Progress
Privacy:  Public Assigned to:  _71007
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 09 Dec 2014 10:56:36 PM UTC, comment #3: 

Hi Werner,

Thank you for the detauled reply.

A few thoughts:

This was tested with version 1.5.2.

I understand how setting delay_upload to 1 would prevent errors being propagated back to the application.  This is a non-trivial danger in some situations, and should be carefully documented.  That, however, wasn't the case here since delay_upload was 0.

I am unsure why disabling locks would prevent an error from being returned due to the unsuccessful creation of a new file.

The corruption was, I believe, not corruption of a successfully-uploaded file but rather a file that was indicated to the application layer that it was uploaded successfully but didn't appear on the server at all.

I did find files in lost+found.  Indeed they were missing Obnam chunks.  However, it appears the failure was not indicated back down to the application.  (Perhaps because I had disabled locks.)

It also should be very carefully documented that disabling locks disables returning errors back to applications.  This is a key issue (and hopefully could be addressed in the future).

Thanks!

John

John Goerzen <jgoerzen>
Tue 09 Dec 2014 07:49:00 PM UTC, comment #2: 

I/O-errors:
HTTP does not know incremental uploads. All writes to a file in davs2 will go into the cache and only when the file is closed davfs2 will upload the file.
Because of this network problems can only cause an I/O-error when

  • You open an existing file
  • you create a new file and use locks
  • you close a file with option "delay_upload 0" set.

But all versions up to 1.5.0 have a bug and don't return an I/O-error on close even with delay_upload 0. This bug is fixed in version 1.5.1 (which is in Debian Testing).
I don't know whether I/O-errors on close will  be of much help. Applications might just ignore them.

Corrupted files:
It is not clear to me. My assupmption: There are your files that you want to back up. Davfs2 will probably never see this files.
There are the "chunks" created by Obnam. This will be the files seen by davfs2.

With this assumption:
davfs2 will upload files with HTTP-method PUT. It will only consider an uplad as successfull when it gets a proper staus code back from the server. Network timeouts will be considered as temporary failures. In this case davfs2 will just hold the file in the cache and retry uploading with increasing time intervals. This also works if the file system is unmounted and mounted again.
But when uploading fails more often than "max_upload_attempts" times then davfs2 will consider this a permanent failure. The file will then be moved into the lost+found directory.
Most probably the corrupted files are caused by missing Obnam chunks. You should find these chunks, with some random string appended, in lost+found.

To increase the chance to notice network failures you may

  • use locks even when you don't need locking
  • install davfs2 version 1.5.1 or 1.5.2


To increase the chance that files don't go into lost+found but are uploaded after the network is operable again:

  • increase max_upload_attempts
  • increase max_retry
  • be patient.


Other reasons for corrupted files:
Corruption may also be caused by errors in the upload process. But this is not very likely and I would only consider it when other explanation prove wrong.

General remark:
I still think that davfs2 is usefull to allow applications without support for WebDAV to access WebDAV-resources like a regular file system.
But this will never be as good as a "real" file system. There is simply no 1:1 relation between the WebDAV-protocol and file system operations.
Now I notice that many people try to use some Webspace for backups, using davfs2. You cold make more of WebDAV if the Backup application would use WebDAV directly without that imperfect file system layer in between. Maybe the developer of Obnam is open for that idea. (With some help, but at least for the next year I am not able to do any serious work on this). The Neon library used by davfs2 was - as far as I know - intended for use cases like this.

Werner

- <_71007>
Sat 06 Dec 2014 01:16:30 PM UTC, comment #1: 

One additional comment - there was never an I/O error returned to the application during this.

John Goerzen <jgoerzen>
Sat 06 Dec 2014 01:13:12 PM UTC, original submission:  

Hello,

Yesterday, I had davfs2 mounted and had an Internet outage that lasted approximately two hours.  During that outage, files of approximately 1MB each were being uploaded.  I observed, surprisingly, that even though the Internet was down, these uploads appeared to occasionally go through.

I figured perhaps this was due to caching, but later I discovered that there was indeed some corruption in the files that were being uploaded just then.  It is hard to define specifically what the corruption was (obnam was doing a backup, and it has "chunks" -- I am looking into it.)

My config was mostly the default, with these possibly-relevant changes:

max_retry 100
connect_timeout 60
read_timeout 60
use_locks 0   (this will only be accessed by one process at a time)
delay_upload 0
minimize_mem 1

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by jgoerzen (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-12-09 _71007 StatusNone In Progress
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-ec04.
    Corresponding source code