helpdavfs2 - Support: sr #109587, delay_upload without closing file

 
 

sr #109587: delay_upload without closing file

Submitted by:  None
Submitted on:  Fri 16 Nov 2018 05:59:54 AM UTC  
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  Werner Baumann <wbaumann>
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Thu 17 Jan 2019 05:49:47 PM UTC, comment #17: 

Thanks to Nikita for asking the right question.

When a davfs2 file system is unmounted then davfs2 writes an index file. This files contains all the metadata necessary to reuse the files in the cache after the next mount. In case of an unclean shutdown this index file can't be written and on the next mount the old index file is used which may not be consistent with the state of the cache.

Ther are two important cases:

  • For a cache file no metatdata are in the old index file: this cache file is moved into the lost+found directory. No data is lost.
  • There is a entry in the index for the cache file but metadata of the file are different from the index: the cache file is deleted. This is a bug (no idea what I thought when I created this many years ago). This file must be moved to lost+found too.

This bug is no fixed in the CVS-sources. I think there will be a new release soon that contains this bugfix.

Please note:

  • This bugfix does not turn davfs2 into a reliable bachup solution. It is not and it won't be.
  • This bug is not related to the original request (changing upload strategy).
  • Check your lost+found directory regularly.

And finally fo all users of LibreOffice:
Use the built-in support for WebDAV. It should do a better job than davfs2. If it does not: the LibreOffice team is waiting for your bug reports.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Thu 17 Jan 2019 07:35:49 AM UTC, comment #16: 

Is davfs expected to re-take file from cache in local filesystem and not from server after unclean shutdown?

Maybe problem is that it does not do that for some reason?

Nikita Yushchenko <nikita_yoush>
Thu 17 Jan 2019 07:31:59 AM UTC, comment #15: 

davfs2 is not a "cloud"-service, it is not a networked-backup, it never promised to be one and it will not be.

Because there is always some change of a network failure davfs2 can't guarantee to upload the file. And because davfs2's interaction with the user is restricted by the file system interface it can't inform the user about that kind of problem. Therefore davfs2 can't be a reliable bauckup. The best it can do is to keep all changes that are not uploaded to the server in the cache. That is what davfs2 does, except when someone destroys files on the local drive. (By the way: what kind of file system do you use that can't resume from a shutdown with open files?)

".. but the use-case described is very common in corporate Linux desktop office environments."
If this happens in a corporate environment then you have to blame the administrator who makes data security depend on pure assumption he never checked. For corporate environments you might consider a real network file system and some kind of automatic backup (it is not as cheap as free space on a WebDAV-server).

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Thu 17 Jan 2019 06:15:07 AM UTC, comment #14: 

> I realize that from technical point of view current behavior may be  "correct" behavior (file is kept open thus it could be written to soon thus let's not flush it)


"current behavior may be "correct"" – actually not. The way Libreoffice (and most other apps) work is to write the file, close it, then open it again and keep it open without changes until the moment the user saves new changes (and then the cycle is repeated). So there are no "pending" writes there / partial flush.

The current behavior with 'delay_upload' > 0 is to check, after the file was written and closed, whether the file is still closed after the delay. This check is a general condition in the upload logic, as a result of code reuse, without taking into account this scenario.

IMO, this is actually an unintended consequence of the implementation, because without the 'delay_upload' option (or when it is set to '0'), after the WRITE + CLOSE the file is sent to the server immediately, without worrying about a partial flush, so why should we worry about a partial flush if it was opened again but no new writes were made – just a mere OPEN operation?

The proposed patch saves to the server the file that was written and closed, even if it was opened again later without new writes. If there were new writes to the file without closing it, it won't save it to the server, so there's no risk of writing a partially saved file.

The logic is the following:
1. If the file was modified (WRITE event) => wait for CLOSE event
2. CLOSE event detected => start counting for the delay_upload seconds
3. Counter reaches 0
3.1 there were NO new WRITEs (changes to the file) during the delay => save the file to the server
3.2 there were new WRITEs (changes to the file) during the delay => cancel the upload and goto 1 (i.e. wait for CLOSE event)

Anonymous
Thu 17 Jan 2019 05:39:53 AM UTC, comment #13: 

Saving to cloud is typically considered a backup replacement - because clouds do provide reliable storage, versioning, etc.

davfs very looks like a tool to (transparently) save to cloud.

Thus the fact that "save" action in Libreoffice for a file on davfs does not save to cloud, is quite unexpected for user.

I'm pretty sure that right now there are quite a few users over the world who saved their work to files on davfs and are pretty sure that their data is safely stored in datacenters - while in fact all that data is in davfs cache only and random system crash of poweroff will destroy it.

I'd not underestimate this effect.

I realize that from technical point of view current behavior may be  "correct" behavior (file is kept open thus it could be written to soon thus let's not flush it).  But this does put users under danger of data loss.

Nikita Yushchenko <nikita_yoush>
Thu 17 Jan 2019 04:24:36 AM UTC, comment #12: 

All you say about backup is valid, but the use-case described is very common in corporate Linux desktop office environments. The users open the documents from WebDAV server via davfs2 and work with them all day long, periodically saving them without closing.

So why to cripple on purpose davfs2 in the name of not introducing the 39th option (yes, today davfs2 has 38 options in the /etc/davfs2/davfs2.conf) as proposed before? The patch is just 19 lines of trivial changes, doesn't have side effects and solves a really severe issue for those who use it as described.

Anonymous
Wed 16 Jan 2019 07:42:23 PM UTC, comment #11: 

It is not a good idea to have the result of a whole day of work in one single (and open) file.

What happened is that a failure destroyed files on a disk. It is of no importance that this file was a davfs2 cache file. If Libreoffice had stored the file directly on the local file system then the result would have been the same (it is Libreoffice that holds the  file open). If the file is on a remote WebDAV server and is destroyed then data is lost too.

davfs2 is not a security backup system. To be save in case of a system failure you need some independent backup of your work.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon 14 Jan 2019 07:27:36 AM UTC, comment #10: 

A person here just lost a week of work due to this behavior of davfs.

She was working on a large libreoffice text document, saving periodically but keeping window open. Then her computer got a power failure. After restart, libreoffice opened a very old copy of document. By server log, davfs never tried to upload document to server since then. And cached data was likely lost due to unclean shutdown.

That level of unrealiability of davfs+libreoffice combination really surprised me.  User naturally expect that "Save" action does save the work. Keeping "saved" work in cache for days is quite undesirable behavior. If davfs has such misfeature then at least a very loud warning should be put at very visible location.

Nikita Yushchenko <nikita_yoush>
Fri 28 Dec 2018 09:38:04 AM UTC, comment #9: 

I think this use case is to special while davfs2 has already too much configuration options.So tha answer is no.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 05 Dec 2018 02:58:08 AM UTC, comment #8: 

What about introducing a new config option for this behavior? It could be off by default so nobody is impacted upon deployment of a new version.

If I provide a complete patch with the new option, would you consider incorporating it?

Anonymous
Tue 04 Dec 2018 06:14:09 PM UTC, comment #7: 

The patch may be well suited for your demands. But other peaple might not be that happy about the increased amount of data transfer.

I think I should not optimize davfs2 for this special use case.

Werner

Please remember: WebDAV is best used from within the application. LibreOffice has built-in WebDAV-support and you could help to make it better (and help to fix that most probably broken WebDAV-server).

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 27 Nov 2018 08:48:16 PM UTC, comment #6: 

What do you think about the attached patch?

(file #45542)

Anonymous
Tue 27 Nov 2018 07:44:17 PM UTC, comment #5: 

OK, I now see what prevents the file to be uploaded.

The dav_tidy_cache function checks the file upload time (save_at) and if it is less than now (time(NULL))... it also checks whether the file is currently opened for write (!is_open_write(node)).

But in LO case (and many other editors) the file is opened again after saving and closing, so it's not uploaded. The previous diagram now looks this way:

MODIFY test.xlsx (more than 50 times)
CLOSE_WRITE,CLOSE test.xlsx <====== here delay_upload=5 starts counting
OPEN test.xlsx
CLOSE_WRITE,CLOSE test.xlsx <====== here the counter is updated
OPEN test.xlsx
<====== BUT here the file is not uploaded 5 seconds after the 2nd CLOSE, because it was again opened for writing by the last OPEN

Why can't we upload a new version of a file while it's opened for writing? To make it perfectly clean we could check if the file had any MODIFYs after the last close (i.e. pending changes) and not upload it in this case, but if it's just opened for writing without any outstanding changes, I guess it could be saved without issues.

Anonymous
Tue 27 Nov 2018 06:42:23 PM UTC, comment #4: 

Werner,

How do you detect that the file is open?

From the inotifywait log it appears that LO closes the file, then opens it again. Even the lock is released on server when LO saves the file (without closing it from within LO). So why "delay_upload > 0" doesn't upload files with a delay at this moment?

Shouldn't it be triggered with the CLOSE operation?

MODIFY test.xlsx (more than 50 times)
CLOSE_WRITE,CLOSE test.xlsx <====== here I'm expecting delay_upload=5 to start counting
OPEN test.xlsx
CLOSE_WRITE,CLOSE test.xlsx <====== here the counter would be reset (this happens during the same second)
OPEN test.xlsx

<====== and here it would finally upload the file, 5 seconds after the 2nd CLOSE

Why it's not working this way?

Anonymous
Tue 27 Nov 2018 05:34:26 PM UTC, comment #3: 

"For me the internal WebDAV client in LibreOffice (LO) just doesn't work, it shows the folders, but can't open any folder that has files, so I'm not even considering it."
I don't like this statement. You really should try to fix this problem with the people from LibreOffice and Cyrus.

davfs2 is restricted by the file system interface. It can't be optimized for the special needs of single Applications. But LibreOffice can.

davfs2 will not upload any files that are open. It is impossible to know from the file system interface when this would be useful and when this will be a horrible waste of resources. So davfs2 will not do this.

You could set "delay_upload 0": this will get your changes to the server on every "save" but will be a lot of useless uploads.

Or you can avoid useless uploads. Then the only time the document is written is when you close it in LibreOffice (the reason is that LibreOffice seems to keep the file open).

This is a LibreOffice problem that can't be resolved by davfs2.

The WebDAV implementation of LibreOffice works with most servers. So the first thing you should do is to check what is going wrong with your Cyrus server.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 21 Nov 2018 04:00:59 AM UTC, comment #2: 

Hi Werner,

For me the internal WebDAV client in LibreOffice (LO) just doesn't work, it shows the folders, but can't open any folder that has files, so I'm not even considering it. By the way, the WebDAV server is Cyrus 3.0.5.

Davfs2 works fine for me, the only issue is that when I use the delay_upload option with a positive value the changes are not uploaded until I close the document in LO. So, if I keep it open all the day, making changes and hitting save, the server never receives the new version of the document. The document is uploaded only when I close it in LO.

Without this option the file is uploaded 3 times on every LO save operation, blocking LO during this process.

I do understand the internals of the operations on files. In this case LO makes a couple of CLOSEs on save, but for some reason with a positive delay_upload they are not taken into account.

Here is an example of what is reported by inotifywait when I save the document in LO (the watch is on the mounted folder):

MODIFY test.xlsx (more than 50 times)
CLOSE_WRITE,CLOSE test.xlsx
OPEN test.xlsx
CLOSE_WRITE,CLOSE test.xlsx
OPEN test.xlsx

And here are the operations when I close the document in LO (already saved with the previous operation):

CLOSE_WRITE,CLOSE test.xlsx
OPEN .~lock.test.xlsx#
ACCESS .~lock.test.xlsx#
CLOSE_NOWRITE,CLOSE .~lock.test.xlsx#
DELETE .~lock.test.xlsx#

I'm not sure how you detect that the file is still open in the editor, but from the inotify events it appears that the file is written, than closed, than opened again on every save in LO.

So what I was thinking is to change the behavior of delay_upload to upload the file on CLOSE event as reported by inotify. Alternatively, a new option could be added if the current behavior of delay_upload is the way it is due to some special reason.

I'm attaching the inotify logs for each operation in LO (e.g. open, type something & save, type something & save again, close) in a separate file + the same operations on a local folder.

Regards,
Anatoli

(file #45504)

Anonymous
Tue 20 Nov 2018 11:22:06 AM UTC, comment #1: 

What "file operations" you do and see within your application (e.g. LibreOffice) and what operations davfs2 sees (upcalls from the kernel) are quite different. davfs2 will never see any Save operation, there is no upcall from the kernel like this.

What davfs2 gets from the kernel are essentially commands like open, close, read, write, create file, delete file. davfs2 cannot know whether the application justs wants to save the current state of the document or wants to stop editing. davf2 just gets: open, write ... write, close. Thats all.

davfs2 only can upload complete files (this is a limit by HTTP) and therefore only uploads file after a close (when no application holds the file open).

If your application closes the file several times within short time, then you can use option delay_upload. davfs2 waits for that long before it starts uploading after close.

One last remark:
WebDAV was not intended to be a file system. The initial intention was that applications have built in support for WebDAV. davfs2 was only intended for applications which can't access WebDAV-resources directly. LibreOffice has WebDAV-support and it should be able to better use WebDAV than a file system. I don't know whether it is really better. But you should give it a try.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 16 Nov 2018 05:59:54 AM UTC, original submission:  

Hi, first of all, thanks for this great software!

I've seen just one issue with a common use-case: some apps (notably LibreOffice Calc) save the same file multiple times during one logical save operation, i.e. when I press Ctrl + S in Calc, it writes the file 3 times! As a result davfs2 makes 3 uploads to the server.

When saving a file of some 5Mb, 3 uploads make the entire process of saving it extremely slow. The solution could be to make delay_upload > 0 to wait for some amount of time to start an upload, but without the requirement to close the file.

The issue with closing files is that when the users edit some shared xls file, they expect others to be able to see the changes (in read-only mode) without having to close and reopen the file on each save.

There could be another option like delay_upload_on_save for this, so when some file is written, davfs2 waits for delay_upload_on_save seconds before considering to upload the changes. If during this time another modify operation occurs on the same file, the wait time is reset and davfs2 waits for delay_upload_on_save again. Once the wait time completes, davfs2 uploads the latest modification to the server.

Regards,
Anatoli

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45542:  pending_changes.patch added by None (2KiB - text/x-patch)
file #45504:  dav_log.tar.gz added by None (818B - application/gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by nikita_yoush (Posted a comment)
  • -email is unavailable- added by nvir
  • -email is unavailable- added by wbaumann (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-17 wbaumann Open/ClosedClosed => Open
    2019-01-17 wbaumann Open/ClosedOpen => Closed
    2018-11-27 None Attached File- => Added pending_changes.patch, #45542
    2018-11-21 nvir Carbon-Copy- => Added nvir
    2018-11-21 None Attached File- => Added dav_log.tar.gz, #45504
    2018-11-20 wbaumann Severity4 - Important => 3 - Normal
    2018-11-20 wbaumann StatusNone => Done
        Assigned toNone => wbaumann

    Back to the top


    Powered by Savane 3.5