helpdavfs2 - Support: sr #108018, Lost updates from multiple clients

 
 

sr #108018: Lost updates from multiple clients

Submitted by:  Jens Lechtenbörger <lechten>
Submitted on:  Mon 09 Apr 2012 02:06:01 PM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 3 - NormalStatus: Postponed
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Open/Closed: OpenOperating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 22 Apr 2012 04:26:38 PM UTC, comment #8:

Please don't get me wrong. I don't suggest not to use WebDAV-locks for better error codes. My suggestion is to offer a POSIX compliant locking mechanism for applications that have been (or will be) written against that standard.

For the time being I'll add code to check whether I'm on a davfs2 filesystem. If so, if open fails, then I'll retry periodically to simulate the blocking call to fcntl(2).

Thanks
Jens

Jens Lechtenbörger <lechten>
Sat 21 Apr 2012 08:39:00 AM UTC, comment #7:

As far as I know only few programs request locks. If davfs2 will use WebDAV-locks only when requested by an application this will mean to not use WebDAV-locks most of the time.

At the moment davfs2 uses WebDAV-locks by default whenever a file is opened for reading. Though WebDAV-locks are far from perfect I still think this the right thing to do. It will prevent most of the cases where concurrent acces is only detect when it is to late, i.e. the client has already changed the file but can not upload the file because it has been changed concurrently.

I dont't think it a good idea to not use WebDAV-locks most of the time just to get a somehow better matching error code.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Thu 19 Apr 2012 06:38:27 PM UTC, comment #6:

I meant advisory POSIX locking via fcntl(2) in FUSE,
int(* lock)(const char , struct fuse_file_info , int cmd, struct flock *),
which was introduced in version 2.6:
http://fuse.sourceforge.net/doxygen/structfuse__operations.html

There are different functions for POSIX locking under Linux, and fcntl(2) is the one that is supposed to work for network filesystems (in contrast to flock(2)). For general information concerning the different locking approaches, I recommend:
http://apenwarr.ca/log/?m=201012#13

With davfs2, I can either disable locking (use_locks 0) or enable mandatory locking (use_locks 1). What I would like to use is advisory fcntl(2) locking (say, use_locks 2). From the fcntl(2) man page:
The l_type field can be used to place a read (F_RDLCK) or a write
(F_WRLCK) lock on a file. Any number of processes may hold a read lock
(shared lock) on a file region, but only one process may hold a write
lock (exclusive lock). An exclusive lock excludes all other locks,
both shared and exclusive.

To acquire a lock there are two options, F_SETLKW and F_SETLK. In case of a locking conflict F_SETLKW waits until the conflicting lock is released while F_SETLK returns immediately with EACCES or EAGAIN. In my previous message I was referring to F_SETLKW.

By implementing the above mentioned FUSE lock function, you could translate calls for fcntl locks into WebDAV locks and return the correct errno from the lock function (instead of returning an error from open)...

Jens Lechtenbörger <lechten>
Wed 18 Apr 2012 07:44:25 PM UTC, comment #5:

'I guess the cleanest solution is to implement lock(), which would block in case of conflicts.'

Could you please explain. I have no idea what you mean.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Sun 15 Apr 2012 05:31:47 PM UTC, comment #4:

> Any idea about a file system error code that better matches this case?


I don't think there is one for open(). I guess the cleanest solution is to implement lock(), which would block in case of conflicts. I don't know how many current users would dislike such a change but I would vote for it ;)

Many thanks for your quick and detailed responses.
Jens

Jens Lechtenbörger <lechten>
Sat 14 Apr 2012 07:35:44 PM UTC, comment #3:

' Using lock_owner does help. Then I get permission denied errors for open() on B as long as the file is open for writing on A. I assume that's davfs2's way of indicating a locking conflict. Is that correct?'
Yes. But I am not happy with this error messages. As a file system davfs2 is bound to use file system error codes and I could not find anything that really matches the case of a locked resource.
Any idea about a file system error code that better matches this case?

'Still, I'm worried about the fact that appending without lock_owner leads to intermediate errors that reduce the file (empty or missing)... '
Well, the WebDAV specs claim to solve the lost update problem with the help of locks. But this does not always work. Beeing able to know the owner of the lock helps.
If you connect to the same server with the same username with the same kind of client more than once at the same time you have concurrent access. But its kind of self concurrent. It is up to you to make it possible to distinguish the two concurrently accessing clients by setting lock_owner.
But still no update is lost. When a resource is changed on the server while it is at the same time edited locally, davfs2 will usually detect this and store the local changes in lost+found. You will find the lost updates there and are free to send them to the server (overriding the other update) or merge them with the other update. But you must resolve that conflict ba hand.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Fri 13 Apr 2012 05:43:35 PM UTC, comment #2:

That you for your detailed answer. Using lock_owner does help. Then I get permission denied errors for open() on B as long as the file is open for writing on A. I assume that's davfs2's way of indicating a locking conflict. Is that correct?

Still, I'm worried about the fact that appending without lock_owner leads to intermediate errors that reduce the file (empty or missing)...

Thanks
Jens

Jens Lechtenbörger <lechten>
Wed 11 Apr 2012 07:42:34 PM UTC, comment #1:

There are different problems acting together.

1. Locks
WebDAV locks are not as reliable as one may think. Because of the problem of stale locks a client can retrieve the lock-token from the server and use it. davfs2 makes use of this. It tries to check wether it is the owner of the lock or someone else.
As you use two davfs2 clients that connect to the server using the same username the lock-owner is the same. In cases like this you must configure option 'lock_owner' with different values for each client.
This will prevent client B from steeling the lock of client A as it happens in your example. (This is the main source of the trouble.)

2. Synchronization:
With a remote repository client and server will not all the time be in sync. This is especially true with caching. And davfs2 needs some degree of caching, otherwise performance with WebDAV would be horrible (for a file system).
So when B deletes a file and client A deletes the same file immedeatly after this client A does not yet know that the file has gone. So both deletes will succeed. I don't think this a problem. For both clients it looks like this:
- they have file foo
- they delete file foo with success
- file foo has vanished
If Client A and client B communicate directly they don't need WebDAV and davfs2. It is important that the file system on A is consistent for the users on system A and the same for B.
It would be confusing for a user on system A if
- file fooo exists
- the user deltes file foo, but delete fails.

3. Upload delays:
davfs2 will upload changed and new files only after the file is closed (there is no way around this). But additionally davfs2 will delay the upload for some seconds. This is to avoid uploading temporary files that will be removed as soon as they are transfered to the server. You can avoid this delay by setting option 'delay_upload 0'. You should only do this when you really need it because it may create a lot of useless traffic and the file system will be less responsive.

Cheers
Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon 09 Apr 2012 02:06:01 PM UTC, original submission:

I'm using davfs2-1.4.6 (with default settings) and apache-2.2.22.
I'd like to append to a single file from multiple locations, but I'm experiencing lost updates. I'm mounting a single WebDAV directory (using the same WebDAV user) on two different client machines, called A and B below.

I've got two test sequences, the first one with shell commands, the second one with a combination of shell and Python to see more details.

First test sequence:
On A:
$ echo "Hallo" >> foo

Then, on B:
$ echo "Moin" >> foo
bash: foo: No such file or directory
$ echo "Moin" >> foo

Then, on A:
$ cat foo
cat: foo: No such file or directory
$ cat foo
cat: foo: No such file or directory
$ cat foo
Moin
$ rm foo

Then, on B:
$ rm foo

In the above sequence, why does the second append on B succeed if the first one fails? In addition, "No such file or directory" is a pretty surprising error message for a command that should create the file if it does not exist. Finally, why can I delete twice?

Apache logs for above commands:
A - jens [09/Apr/2012:15:37:44 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1249
A - jens [09/Apr/2012:15:37:44 +0200] "LOCK /webdav/foo HTTP/1.1" 200 418
B - jens [09/Apr/2012:15:37:46 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
B - jens [09/Apr/2012:15:37:46 +0200] "LOCK /webdav/foo HTTP/1.1" 423 277
B - jens [09/Apr/2012:15:37:46 +0200] "PROPFIND /webdav/foo HTTP/1.1" 207 587
B - jens [09/Apr/2012:15:37:46 +0200] "LOCK /webdav/foo HTTP/1.1" 200 418
B - jens [09/Apr/2012:15:37:46 +0200] "GET /webdav/foo HTTP/1.1" 404 208
B - jens [09/Apr/2012:15:37:46 +0200] "UNLOCK /webdav/foo HTTP/1.1" 204 -
B - jens [09/Apr/2012:15:37:46 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1249
B - jens [09/Apr/2012:15:37:48 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1249
B - jens [09/Apr/2012:15:37:48 +0200] "LOCK /webdav/foo HTTP/1.1" 200 418
A - jens [09/Apr/2012:15:37:54 +0200] "PUT /webdav/foo HTTP/1.1" 412 244
A - jens [09/Apr/2012:15:37:54 +0200] "UNLOCK /webdav/foo HTTP/1.1" 400 226
A - jens [09/Apr/2012:15:37:54 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
A - jens [09/Apr/2012:15:37:56 +0200] "GET /webdav/foo HTTP/1.1" 404 208
A - jens [09/Apr/2012:15:37:57 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
A - jens [09/Apr/2012:15:37:59 +0200] "GET /webdav/foo HTTP/1.1" 404 208
B - jens [09/Apr/2012:15:37:59 +0200] "PUT /webdav/foo HTTP/1.1" 201 183
B - jens [09/Apr/2012:15:37:59 +0200] "HEAD /webdav/foo HTTP/1.1" 200 -
B - jens [09/Apr/2012:15:37:59 +0200] "UNLOCK /webdav/foo HTTP/1.1" 204 -
A - jens [09/Apr/2012:15:38:00 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1730
A - jens [09/Apr/2012:15:38:01 +0200] "GET /webdav/foo HTTP/1.1" 200 5
A - jens [09/Apr/2012:15:38:15 +0200] "DELETE /webdav/foo HTTP/1.1" 204 -
B - jens [09/Apr/2012:15:38:17 +0200] "DELETE /webdav/foo HTTP/1.1" 404 208
B - jens [09/Apr/2012:15:38:17 +0200] "PROPFIND /webdav/foo HTTP/1.1" 404 208
B - jens [09/Apr/2012:15:38:18 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1249

Here is a similar test sequence using Python commands (preceded by >>>) and shell commands (preceded by $):
On A:

>>> import os
>>> f = open("foo", "a")


On B:

>>> f = open("foo", "a")

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: 'foo'

So, open() fails. Apart from the error message itself, which does not hint towards locking conflicts, this might be fine. However, repeating the command succeeds:
On B:

>>> f = open("foo", "a")


On A:

>>> f.write("Moin\n")
>>> f.flush()
>>> os.fsync(f.fileno())
>>> f.close()


On B:

>>> f.write("Hello\n")
>>> f.close()


On B:
$ cat foo
Hello

On A:
$ cat foo
cat: foo: No such file or directory
<The previous message occurs for several invocations of cat.>
$ cat foo
<empty result>
<wait more>
$ cat foo
Hello
$ rm foo
<success, no output>

On B:
$ rm foo
<success, no output>

Thus, why does the second open() on B succeed if the first one fails? Then, all writes of A are lost (although I'm trying to sync explicitly), which is bad. Moreover, in between the file seems to disappear on A, which is even worse.
Then, why does deletion succeed twice?

Apache logs for Python tests:
A - jens [09/Apr/2012:15:31:37 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1249
A - jens [09/Apr/2012:15:31:37 +0200] "LOCK /webdav/foo HTTP/1.1" 200 418
B - jens [09/Apr/2012:15:31:39 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
B - jens [09/Apr/2012:15:31:39 +0200] "LOCK /webdav/foo HTTP/1.1" 423 277
B - jens [09/Apr/2012:15:31:39 +0200] "PROPFIND /webdav/foo HTTP/1.1" 207 587
B - jens [09/Apr/2012:15:31:39 +0200] "LOCK /webdav/foo HTTP/1.1" 200 418
B - jens [09/Apr/2012:15:31:39 +0200] "GET /webdav/foo HTTP/1.1" 404 208
B - jens [09/Apr/2012:15:31:39 +0200] "UNLOCK /webdav/foo HTTP/1.1" 204 -
B - jens [09/Apr/2012:15:31:39 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1249
B - jens [09/Apr/2012:15:31:41 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1249
B - jens [09/Apr/2012:15:31:41 +0200] "LOCK /webdav/foo HTTP/1.1" 200 418
A - jens [09/Apr/2012:15:31:48 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
B - jens [09/Apr/2012:15:32:02 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
A - jens [09/Apr/2012:15:32:07 +0200] "PUT /webdav/foo HTTP/1.1" 412 244
A - jens [09/Apr/2012:15:32:07 +0200] "UNLOCK /webdav/foo HTTP/1.1" 400 226
A - jens [09/Apr/2012:15:32:08 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
A - jens [09/Apr/2012:15:32:08 +0200] "GET /webdav/foo HTTP/1.1" 404 208
A - jens [09/Apr/2012:15:32:09 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
A - jens [09/Apr/2012:15:32:10 +0200] "GET /webdav/foo HTTP/1.1" 404 208
A - jens [09/Apr/2012:15:32:10 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
A - jens [09/Apr/2012:15:32:12 +0200] "GET /webdav/foo HTTP/1.1" 404 208
A - jens [09/Apr/2012:15:32:12 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1666
B - jens [09/Apr/2012:15:32:13 +0200] "PUT /webdav/foo HTTP/1.1" 201 183
B - jens [09/Apr/2012:15:32:13 +0200] "HEAD /webdav/foo HTTP/1.1" 200 -
B - jens [09/Apr/2012:15:32:13 +0200] "UNLOCK /webdav/foo HTTP/1.1" 204 -
A - jens [09/Apr/2012:15:32:14 +0200] "GET /webdav/foo HTTP/1.1" 200 6
A - jens [09/Apr/2012:15:32:16 +0200] "GET /webdav/foo HTTP/1.1" 304 -
A - jens [09/Apr/2012:15:32:19 +0200] "DELETE /webdav/foo HTTP/1.1" 204 -
B - jens [09/Apr/2012:15:32:22 +0200] "DELETE /webdav/foo HTTP/1.1" 404 208
B - jens [09/Apr/2012:15:32:22 +0200] "PROPFIND /webdav/foo HTTP/1.1" 404 208
B - jens [09/Apr/2012:15:32:23 +0200] "PROPFIND /webdav/ HTTP/1.1" 207 1249

Jens Lechtenbörger <lechten>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by lechten (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 20 Apr 2014 07:09:19 PM UTCwbaumannStatusIn Progress=>Postponed
    Wed 11 Apr 2012 07:42:34 PM UTCwbaumannSeverity4 - Important=>3 - Normal
      StatusNone=>In Progress
      Assigned toNone=>wbaumann

    Back to the top


    Powered by Savane 3.1-cleanup