helpdavfs2 - Support: sr #108697, Corruption on upload during...

 
 

sr #108697: Corruption on upload during extended outage

Submitted by:  None
Submitted on:  Sat Dec 6 13:13:12 2014  
 
Category: NonePriority: 5 - Normal
Severity: 4 - ImportantStatus: In Progress
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Originator Email: -unavailable-Open/Closed: Open
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue Dec 9 22:56:36 2014, comment #3:

Hi Werner,

Thank you for the detauled reply.

A few thoughts:

This was tested with version 1.5.2.

I understand how setting delay_upload to 1 would prevent errors being propagated back to the application. This is a non-trivial danger in some situations, and should be carefully documented. That, however, wasn't the case here since delay_upload was 0.

I am unsure why disabling locks would prevent an error from being returned due to the unsuccessful creation of a new file.

The corruption was, I believe, not corruption of a successfully-uploaded file but rather a file that was indicated to the application layer that it was uploaded successfully but didn't appear on the server at all.

I did find files in lost+found. Indeed they were missing Obnam chunks. However, it appears the failure was not indicated back down to the application. (Perhaps because I had disabled locks.)

It also should be very carefully documented that disabling locks disables returning errors back to applications. This is a key issue (and hopefully could be addressed in the future).

Thanks!

John

John Goerzen <jgoerzen>
Tue Dec 9 19:49:00 2014, comment #2:

I/O-errors:
HTTP does not know incremental uploads. All writes to a file in davs2 will go into the cache and only when the file is closed davfs2 will upload the file.
Because of this network problems can only cause an I/O-error when

  • You open an existing file
  • you create a new file and use locks
  • you close a file with option "delay_upload 0" set.

But all versions up to 1.5.0 have a bug and don't return an I/O-error on close even with delay_upload 0. This bug is fixed in version 1.5.1 (which is in Debian Testing).
I don't know whether I/O-errors on close will be of much help. Applications might just ignore them.

Corrupted files:
It is not clear to me. My assupmption: There are your files that you want to back up. Davfs2 will probably never see this files.
There are the "chunks" created by Obnam. This will be the files seen by davfs2.

With this assumption:
davfs2 will upload files with HTTP-method PUT. It will only consider an uplad as successfull when it gets a proper staus code back from the server. Network timeouts will be considered as temporary failures. In this case davfs2 will just hold the file in the cache and retry uploading with increasing time intervals. This also works if the file system is unmounted and mounted again.
But when uploading fails more often than "max_upload_attempts" times then davfs2 will consider this a permanent failure. The file will then be moved into the lost+found directory.
Most probably the corrupted files are caused by missing Obnam chunks. You should find these chunks, with some random string appended, in lost+found.

To increase the chance to notice network failures you may

  • use locks even when you don't need locking
  • install davfs2 version 1.5.1 or 1.5.2

To increase the chance that files don't go into lost+found but are uploaded after the network is operable again:

  • increase max_upload_attempts
  • increase max_retry
  • be patient.

Other reasons for corrupted files:
Corruption may also be caused by errors in the upload process. But this is not very likely and I would only consider it when other explanation prove wrong.

General remark:
I still think that davfs2 is usefull to allow applications without support for WebDAV to access WebDAV-resources like a regular file system.
But this will never be as good as a "real" file system. There is simply no 1:1 relation between the WebDAV-protocol and file system operations.
Now I notice that many people try to use some Webspace for backups, using davfs2. You cold make more of WebDAV if the Backup application would use WebDAV directly without that imperfect file system layer in between. Maybe the developer of Obnam is open for that idea. (With some help, but at least for the next year I am not able to do any serious work on this). The Neon library used by davfs2 was - as far as I know - intended for use cases like this.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sat Dec 6 13:16:30 2014, comment #1:

One additional comment - there was never an I/O error returned to the application during this.

John Goerzen <jgoerzen>
Sat Dec 6 13:13:12 2014, original submission:

Hello,

Yesterday, I had davfs2 mounted and had an Internet outage that lasted approximately two hours. During that outage, files of approximately 1MB each were being uploaded. I observed, surprisingly, that even though the Internet was down, these uploads appeared to occasionally go through.

I figured perhaps this was due to caching, but later I discovered that there was indeed some corruption in the files that were being uploaded just then. It is hard to define specifically what the corruption was (obnam was doing a backup, and it has "chunks" -- I am looking into it.)

My config was mostly the default, with these possibly-relevant changes:

max_retry 100
connect_timeout 60
read_timeout 60
use_locks 0 (this will only be accessed by one process at a time)
delay_upload 0
minimize_mem 1

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by jgoerzen (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue Dec 9 19:49:00 2014wbaumannStatusNone=>In Progress
      Assigned toNone=>wbaumann

    Back to the top


    Powered by Savane 3.1-cleanup1