maindavfs2 - Support: sr #107919, No support for PUT errors

 
 

sr #107919: No support for PUT errors

Submitter:  Gareth Bult <garethbult>
Submitted:  Fri 23 Dec 2011 01:48:22 AM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Wont Do
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 30 Dec 2011 04:27:55 PM UTC, comment #7: 

"Mmm, in this specific instance the assumption would be incorrect, but IMHO it's a factor that DavFS should not be aware of, indeed it should assume that the underlying storage mechanism does in fact perform as it should and take things from there. "We're not going to work properly because we assume the surrounding systems don't work properly" isn't necessarily an endearing quality in software .. ;)"
You misunderstood what I wanted to say. It is this: Almost every file system today uses caching and there is always a chance that an error occurs after the file system call returned with success. This is nothing davfs2 specific. But with unreliable network connections and buggy servers it is more like to happen than with harddisks.

"Our emphasis is on read speed, write speed is almost irrelevant, so properly syncing write's is a bearable overhead."
Your use of davfs2 looks like a corner case. I have to mainly care about the main intended use of davfs2.

"so properly syncing write's is a bearable overhead."
Not at all, it would be a performance nightmare. Making close() work synchronously would not make write's synchronously. To sync write's every single write request from the kernel would require one HTTP request. This may be many thousends for a single file. To make it bearable at all it would require the client and server to support ranged PUT requests. If one does not support it the file system will be unusable.

By the way:
Do applications care about the return value of close()? And for those that do: how will they react on an error?

If you realy need synchronous write to the server davfs2 will not work. You may take parts of davfs2 but you would have to rewrite most of it from the scratch unless you find another WebDAV file system that better matches your demands. I don't know of any WebDAV file system with sychronous write.

Cheers
Werner

- <_71007>
Fri 30 Dec 2011 01:53:01 PM UTC, comment #6: 


>Well this works almost never.


Mmm, in this specific instance the assumption would be incorrect, but IMHO it's a factor that DavFS should not be aware of, indeed it should assume that the underlying storage mechanism does in fact perform as it should and take things from there. "We're not going to work properly because we assume the surrounding systems don't work properly" isn't necessarily an endearing quality in software .. ;)

Our emphasis is on read speed, write speed is almost irrelevant, so properly syncing write's is a bearable overhead.

Is there any way to make DavFS play ball with the remote server, i.e. do not continue until the server has returned a status, and act on that status in every instance. i.e. if an error is returned, it's an error and not something that will be retried later? I've not checked but from what you've said it sounds like there are instances when this isn't the case.

Just to clarify, I already have this working, performance is more than acceptable and it does everything I want it to do (testing against a number of clients, primarily Windows 7), but I've reached a point where in the interests of data integrity, any client we're going to try to work with needs to accept status returns in cases where operations fail.

I've a couple of other things to work on, then I'm going to be writing a test suite to see how different clients compare to one-another with regards to throughput and forces failures. I'll post some feedback once I have it but if you can think of any other areas that might omit a status return I'd be grateful of the info and any patches.
(I'm going to try the current patch later today)

Many thanks.
Gareth.






Gareth Bult <garethbult>
Fri 30 Dec 2011 11:40:46 AM UTC, comment #5: 

First: applications that use a davfs2 file system don't know about PUT, they use the operating system's file system interface. And , of course, davfs2 checks the status of any PUT request/response.

But comming to the core.
"A "real" filesystem (my terminology) is one where people can store information and be "sure" it's actually been saved."

Well this works almost never. All modern operating systems do caching. When an application "writes to disk" it can not be sure that the data is saved to disk. A disk failure may happen later when the data written are still in the cache in working memory and not on the disk. In this respect it is not really different from davfs2.

Of course there are important differences between data stored on a local harddisk and data stored on some remote server: less reliable network connections, slower connection with greater delays, more bugs in remote servers and independent concurrent acces to the backend store. Operating systems usually don't allow concurrent access to the disk (only the file system has acces to the partition) while with WebDAV concurrency is a necessary feature.

Caching is done to improve performance and reduce the cost (processor time, interface bandwith). But with caching the backend store and what the local applications sees will usally not be in sync. And errors may happen after the local operation hass finished successfully but before the changes are saved to the backend store. This holds true for any kind of caching. There is no "real" solution for this. It is a trade off you can only try to make as smart as possible and to best fit the demands for every special case.

As for WebDAV:
If you want to be sure that the data are stored to the backend store when the file system call returns you must not do any caching at all. Each and every file system call will result in one (often more than one) HTTP request and will only return after the response from the server has been received. This is not only necessary for close() but for every file system call even for these zillions of lookups.
This will decrease performance dramatically and it would be quite another WebDAV client than what davfs2 is supposed to be.

Sync:
With disk file systems applications can use the sync file system call to make sure data are saved to disk. davfs2 could to the same. But I noticed that these sync calls are quite frequent (not only when the application explicitely requests). Therefore davfs2 will on sync not sync to the WebDAV server but only to the local cache on disk.

Handling errors that occur after the file system call returned but before data is stored to the server:
davfs2 can't return an error code in this case (because the file system call already returned). davfs2 tries to distinguish temporary and permanent failures.
In case of a temporary failure davfs2 will hold the data in the cache and retry syncing with the server later. Applications can access the file as usual. Requests are served from the cache. Changes will be synced to the server when the temporary failure has gone.
In case of a permanent error davfs2 will save all local changes in the lost+found directory. For access to the original file davfs2 will have to get it from the server again. If this fails the file system call returns with an error.

Cheers
Werner

- <_71007>
Fri 30 Dec 2011 12:32:55 AM UTC, comment #4: 

Excellent, many thanks I'll give it a go.

A "real" filesystem (my terminology) is one where people can store information and be "sure" it's actually been saved. Without any status checking, you've no idea whether the result of your PUT has actually resulted in your data being stored. On a normal filesystem you only have to deal with very unlikely raw write errors and probably infrequent out of disk space errors - but all the same these things can happen and ignoring them is probably less than ideal. Moreover, as I'm storing in a database there's a whole extra layer of potential database failure to cope with so for me it becomes absolutely critical.

The filesystem I'm running is specifically for content management, in this context thus far it seems to work well and certainly satisfies our criteria re; performance and reliability - the only issue I've had so far re; reliability has been with the lack of Status on DavFS .. hopefully this will sort the issue :)

tvm.


Gareth Bult <garethbult>
Wed 28 Dec 2011 08:25:49 PM UTC, comment #3: 

Attached you will find a patch for cache.c.
With this patch davfs2 will return an error on close if there is a permanent failure to upload a file. You can't expect davfs2 to return an error if the failure is temporary. In this case returning an error would be wrong or davfs2 would be forced to not try the upload again.

I did not really understand what you are doing, Nevertheless here are some remarks:

"Otherwise DavFS2 (IMHO) can't support any sort of "real" filesystem other than just remote mounting remote web-servers for single-user use."
Well, mounting a remote WebDAV-repository into a local file system, that's exactly what davfs2 is for. Any idea how you would map a WebDAV server into what you call a "real" file system?
davfs2 sure is not restricted to "single-user use". A mounted WebDAV file system behaves in most respects just like any other file system. Of course it may be accessed by as many users, processes and threads as your operating system can handle.

"Client;

LOCK
PUT
(wait 3s)
<actual put>
UNLOCK "
No! Regarding the HTTP-traffic it is:
LOCK
GET
PUT
UNLOCK
And there may be a lot of PROPFINDs as well as some HEAD requests. Of course there may be any amount of delays between the different HTTP-requests depending on many factors. The delay due to delay_upload is just one of them and the server can't tell the difference. The protocol does not depend on this.
On the other side. Application accessing the file system don't know of any of this. They use the file system interface.

"Granted the client will yield after the PUT, but any further operations that rely on the PUT will either block on the LOCK, or return the wrong information because although the client has completed it's "write" operation, the server hasn't actually been given the information."
What would you expect from a file system when the "backend" is connected through an unreliable network connection. The connection may break at any time and recover at any time. Client and server have to care about a reasonable handling of these error cases, but they can't avoid them.
Regarding locks: your server should have some timeout for stale locks.
And of course you can't expect that the server is always in sync with the local davfs2 file system if only for the delay on the network. And davfs2 is designed to work with slow and very slow connections as well as with LANs.

"any further operations that rely on the PUT will either block on the LOCK"
I don't understand this. If davfs2 is dead there is no more file system. Processes will get just this information. There are no further operations that might rely on the PUT. Anyway applications that access a davfs2 file system don't know about PUT. They use the system's file system operations. As long as davfs2 is alive these operations will work. If davfs2 can't upload a file due to an temporary failure applications can still access this file.

Concurrency:
When two clients are connected to the same server any client may lock a file and hold the lock for as long as it wants for whatever reason it likes. This is of no interest for the other client. It just gets informed about the lock. What difference would it make for the other client to know whether the first client locks the file because it is still changing the file or because it just waits until it seems time to try an upload?

I have the impression that your second client not only is connected to the WevDAV server but also to the first client and now expects that information it gets from the first client directly is in sync with information from the server. This will not work and it will not work with most other network file systems.

If you have more than one application running that interact with one another and that need to access the same WebDAV resources: just mount the WebDAV resource once with davfs2 and let the applications acces this file system.

One last remark:
If you want to create a fast, reliable and massively scalable network file system - forgett about WebDAV.

Cheers
Werner


(file #24707)

- <_71007>
Wed 28 Dec 2011 01:29:28 PM UTC, comment #2: 

Hi Werner, many thanks for the reply.
First let me say that I have tried a number of status codes, and indeed have taken a look at the DavFS2 source code, hence my submission .. :)

I appreciate how it is working and to an extent why, so maybe it'll help if I explain what I'm doing. Essentially I've written a WEBDAV server that maps storage onto MongoDB, essentially "under the hood" I guess I've created a MongoDBFS filesystem. Now, this works remarkably well both with Linux (DavFS2) and as a storage option for Windows .. indeed (much to my amazement) the combination works as well as Windows drive shares (re performance) but without all the hassle re; remote access and security.

Now, first comment - nothing works unless I set delay_upload to 0, indeed discovering "delay_upload" was a bit of a revelation (!)

Client;

LOCK
PUT
(wait 3s)
<actual put>
UNLOCK

Granted the client will yield after the PUT, but any further operations that rely on the PUT will either block on the LOCK, or return the wrong information because although the client has completed it's "write" operation, the server hasn't actually been given the information. For any sort of shared environment where concurrency is an issue, "delay_upload" is a bit of a "gotcha". i.e. applications that use filesystems expect Async operation, so although caching is nice, Sync operation will break many things that might try to use it.

In order to make this fly (for me);

a. DavFS2 must use delay_upload=0 (obviously this is just a setting, although I submit it would be a good default!)
b. PUT must return a valid success/failure indication - I have experimented with catching errors on UNLOCK, but it's a bodge, and it'll never catch an "out of space" or transient "write" error.

Otherwise DavFS2 (IMHO) can't support any sort of "real" filesystem other than just remote mounting remote web-servers for single-user use.
(and WebDAV it appears can be SO much more)

Consider with MongoDB, the back-end database is replicated and can be scaled to 1000 servers.
Consider also that WebDAV will work through concentrators (I use "varnish") , the scope for a massively scalable filesystem is immense.
- and in testing, I have this working, in terms of response, you wouldn't know on the face of things that it wasn't a local filesystem.

If DavFS2 could support this mode of operation, it would be really cool ... looking at the code it doesn't look like it would take much to return some additional status info .. (?)

Gareth Bult <garethbult>
Wed 28 Dec 2011 11:21:01 AM UTC, comment #1: 

I think there is an error in the status code. "204 Forbidden" is not a valid HTTP status code. As far as I remember - without consulting the spec - it is "204 No Content". All 2xx codes are success-codes. What you need would be "507 Insufficient Space" (please look at the spec to be sure).

But even with the correct status code you will not get a file system error from davfs2. The reason is how davfs2 uses caching to reduce network traffic and increase performance.

When a file is changed locally or created davfs will act like this:

When the file is opened:

  • lock the file on the server (if locks are enabled)
  • download the file into cache if necessary


As long as the file is open:

  • everything is done locally with the cached file


When the file is closed:

  • return success to the file system call
  • wait for delay_upload seconds
  • upload the file
  • if uploading fails: try again some time later in case of a temporary failure; move the file into the lost+found directory in case of a permanent failure.


You may wonder about the delay. Many programs will create temporary files that will be removed immediately after the close. Without this delay they would be send to the server only to be removed immediately after. You can change option 'delay_upload' according your needs. With a value of 0 the file system call will only return after the upload is finished. But in case of a temporary failure (like network timeout) this is not true. The file system call will return with success; the file will stay in the cache and upload will be tried some time later.

The only occasion when davfs2 could meaningfully return an error to the file system's close call would be

  • if delay_upload is 0
  • and the upload fails due to a permanent error.


Currently davfs2 does not do this because I doubt that applications will make use of an error on close(). But if you are interested to test it I could create a patch for this. Please tell me.

Cheers
Werner

- <_71007>
Fri 23 Dec 2011 01:48:22 AM UTC, original submission:  

It seems that although PUT locks an item and that lock failures are reported and handled, 'actual' errors reported by PUT (such as running out of disk space) are ignored.

If I do "cat FILENAME > /mnt/webdav" and /mnt/webdav is a mounted davfs filesystem, I would expect the "cat" command to generate an error if the davfs filesystem fills, currently this is not the case.

I've written a Webdav server in Python for testing purposes and implemented a PUT that simply returns an immediate error. Davfs debug looks like this;

---
Dec 23 01:33:25 scout mount.davfs: Running pre_send hooks
Dec 23 01:33:25 scout mount.davfs: Sending request headers:#012PUT /Fixed/test.txt HTTP/1.1#015#012User-Agent: davfs2/1.4.6 neon/0.29.6#015#012Keep-Alive: #015#012Connection: TE, Keep-Alive#015#012TE: trailers#015#012Host: localhost#015#012If-Match: "UNKNOWN"#015#012Content-Length: 13#015#012Authorization: xxxxxxxxxxxxxxxxxxxxxx#015#012If: <http://localhost/Fixed/test.txt> (<urn:uuid:1b8c820f-2d06-11e1-9c21-e0cb4ecdcb0f>)#015#012#015
Dec 23 01:33:25 scout mount.davfs: Sending request-line and headers:
Dec 23 01:33:25 scout mount.davfs: req: Connecting to 127.0.0.1:80
Dec 23 01:33:25 scout mount.davfs: Sending request body:
Dec 23 01:33:25 scout mount.davfs: Request sent; retry is 0.
Dec 23 01:33:25 scout mount.davfs: [status-line] < HTTP/1.0 204 Forbidden#015
Dec 23 01:33:25 scout mount.davfs: [hdr] Server: AuthHTTP/0.1 Python/2.7.2+#015
Dec 23 01:33:25 scout mount.davfs: Header Name: [server], Value: [AuthHTTP/0.1 Python/2.7.2+]
Dec 23 01:33:25 scout mount.davfs: [hdr] Date: Fri, 23 Dec 2011 01:33:25 GMT#015
Dec 23 01:33:25 scout mount.davfs: Header Name: [date], Value: [Fri, 23 Dec 2011 01:33:25 GMT]
Dec 23 01:33:25 scout mount.davfs: [hdr] Accept-Ranges: bytes#015
Dec 23 01:33:25 scout mount.davfs: Header Name: [accept-ranges], Value: [bytes]
Dec 23 01:33:25 scout mount.davfs: [hdr] DAV: 1,2#015
Dec 23 01:33:25 scout mount.davfs: Header Name: [dav], Value: [1,2]
Dec 23 01:33:25 scout mount.davfs: [hdr] Content-Length: 0#015
Dec 23 01:33:25 scout mount.davfs: Header Name: [content-length], Value: [0]
Dec 23 01:33:25 scout mount.davfs: [hdr] #015
Dec 23 01:33:25 scout mount.davfs: End of headers.
Dec 23 01:33:25 scout mount.davfs: Running post_headers hooks
Dec 23 01:33:25 scout mount.davfs: Running post_send hooks
Dec 23 01:33:25 scout mount.davfs: sess: Closing connection.
Dec 23 01:33:25 scout mount.davfs: sess: Connection closed.
Dec 23 01:33:25 scout mount.davfs: Request ends, status 204 class 2xx, error line:#012204 Forbidden
Dec 23 01:33:25 scout mount.davfs: Running destroy hooks.
Dec 23 01:33:25 scout mount.davfs: Request ends.
---

And although the logs clearly report a "204", my cat command reports no error ...
Any ideas? Am I doing something wrong?

Gareth Bult <garethbult>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by garethbult (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-04-20 _71007 Open/ClosedOpen Closed
    2014-04-20 _71007 StatusIn Progress Wont Do
    2011-12-28 _71007 Attached File- Added davfs2-1.4.6-cache.c-close.diff, #24707
        Severity4 - Important 3 - Normal
    2011-12-28 _71007 StatusNone In Progress
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code