maindavfs2 - Support: sr #107869, uploading files to Xythos server...

 
 

sr #107869: uploading files to Xythos server fails

Submitter:  Bob Tennent <rdtennent>
Submitted:  Fri 11 Nov 2011 02:14:35 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 01 Mar 2012 04:19:19 PM UTC, comment #12: 

sounds like what I wanted to hear!  thanks

Jeff Long <jlong>
Wed 29 Feb 2012 07:03:27 PM UTC, comment #11: 

"I almost have them talked into changing that to use the 200 response".
There should be no need to "talk them into". 201 is a lie and a bug and must be changed into 200 anyway, independend of what davfs2 does.

For lock-null resources (response code 200):

  • davfs2 will never do a GET
  • davfs2 will do no HEAD except you set option 'if_match_bug 1'. If you set 'if_match_bug 1' davfs2 will use HEAD to check that no resource has been created by another client. I expects response code "404 Not Found" in this case before doing a PUT. Even if you set option 'if_match_bug 1' (which is only needed if the server has a bug) you can prevent HEAD by setting option 'precheck 0'.


Note: if the server properly supports "If-Match" and "If-None-Match" davfs2 will avoid most HEAD-requests by using these conditionals.

Cheers
Werner

- <_71007>
Tue 28 Feb 2012 10:11:45 PM UTC, comment #10: 

Can we re-visit this issue?  I've been talking to Xythos about this and doing some investigating.  They are doing the old-style lock-null method from RFC 2518.  That RFC does not allow the 201 response to the LOCK method.  I almost have them talked into changing that to use the 200 response but I would like to know if the davfs2 behavior will change if it sees a 200 response instead of a 201.  i.e. will it only do a PUT as suggested in appendix D of RFC 4918 (no GET/HEAD)?  Or does davfs2 specifically not follow that recommendation?



Jeff Long <jlong>
Wed 28 Dec 2011 11:59:42 AM UTC, comment #9: 

Sorry. Because I was busy I delayed the answer and then I forgott about it.

Here is some background about what is going wrong and why not using locks helps:

The problem is related to how servers must behave when a client requests a lock for a file that does not yet exist.

The original WebDAV specification (RFC 2518) defined:

  • the server must create a lock but not create a file (so only the name is reserved; they called it 'locked Null resource). Status code "200 OK".

At least Microsoft's IIS never complied with this.

When the spec was updated (RFC 4918) a new way was defined (the way IIS handled it):

  • the server must create a new and empty file and lock this file. Status code "201 Created".

Both ways are allowed.

This would not be too bad. davfs2 can handle both ways. But what really fucked it up was that servers started to use any kind of unspecified combination, like responding with 201 Created and not creating a file or responding 200 OK and creating a file. For davfs2 it is important to know whether the file is created or not to be able to do a later PUT in a safe way.

What the Xythos server does seems like such a mismatch. First responding with 201 Created and then responding with 405 Method Not Allowed on HEAD. It did not yet know this variant.

But of course, if you use their own client it will work. Their own client knows of their own varity of the WebDAV protocol.

Cheers
Werner

- <_71007>
Wed 16 Nov 2011 10:08:46 PM UTC, comment #8: 

 > This time the file test.txt already existet on the server and you only changed it.

I was avoiding ls commands as per instructions :+) so I didn't realize it.

> Please use option 'use_locks 0' and try to create a new file. It should work.


It does indeed.  Thank you.  The server is supposed to be working for the Windows Xythos client.  Would it avoid locks?

Bob Tennent <rdtennent>
Wed 16 Nov 2011 08:36:04 PM UTC, comment #7: 

The new log files look fine in all three cases. As I understand your messages the file test.txt was really uploaded to the server. The HTTP-messages in the logs say so. Did you check with another client (browser) to be really sure?
The logs look even better than I expect from most other servers. In response to the PUT-request (file upload) the server sends an Etag. This is a very useful feature which reduces traffic and makes caching mor reliable.

But there is still a problem.
This time the file test.txt already existet on the server and you only changed it. In the old log you created a new file ThisIsATest.
The error happend when dafs2 requested a lock for the non-existing file. The server claimed to have created an empty file but reported an error when davfs2 did an HEAD-request for this file. I asume the problem still exists.

A warkaraound for this problem could be to not use locking. Whether this is feasable depends on how many different clients access the resources at the same time. If there is only your client, locking is not needed at all.
But even if there is concurrent access to the same resources locking is not strictly necessary. davfs2 stores the Etag values and always uses conditional PUT-requests. If someone else changed the file in between this will be detected and davfs2 will not overwrite the changed file on the server. Your local changes will instead end up in the lost+found directory. So you will have to check your lost+found directory if not using locks and if there is danger of concurrent access to the same file (you should check lost+found from time to time anyway).

Because your server supports conditional requests as well as etags you must not use the options 'if_match_bug 1' and 'precheck 0'. The default values of 'if_match_bug 0' and 'preckeck 1' match the behaviour of this server.

Please use option 'use_locks 0' and try to create a new file. It should work.

Werner

- <_71007>
Tue 15 Nov 2011 12:59:08 AM UTC, comment #6: 

The removed lines were produced by sshd, not davfs2.  I'll attach 3 log files:

debug.0  debug most

debug.1  also if_match_bug 1

debug.2  also precheck 0

In every case a text file was uploaded using

echo This is a test. > test.txt

Thanks for your help.

(file #24374, file #24375, file #24376)

Bob Tennent <rdtennent>
Mon 14 Nov 2011 08:25:52 PM UTC, comment #5: 

First:
Don't edit /usr/share/davfs2/davfs2.conf. This are data davfs2 needs unchangd to work properly. Please read 'man mount.davfs' and 'man davfs2.conf'.

Second:
This server seems badly broken. Your logs show some strange behaviour.

Request:
2OPTIONS /Groups/Arts_and_Science/Curriculum/
Response:
HTTP/1.1 200 OK
Allow: OPTIONS, PROPFIND, PROPPATCH, LOCK, UNLOCK, DELETE, GET, HEAD, MOVE, COPY, ACL, SEARCH
DAV: 1,2
Funny. A WebDAV-server (Level 1 and 2) that does not allow PUT (no uploads) but supports LOCK (what for without PUT?). More important: it claims to allow HEAD (more on this later).

Request:
HEAD /Groups/Arts_and_Science/Curriculum/Committee%20Members%20Homework/Sub-Committee%20ABIOL/ThisIsATest
Response:
HTTP/1.1 404 Not Found
That's ok. davfs just checked to make sure it will not overwrite an existing file. It does not yet exist.

Request:
LOCK /Groups/Arts_and_Science/Curriculum/Committee%20Members%20Homework/Sub-Committee%20A/BIOL/ThisIsATest
Response:
HTTP/1.1 201 Created
davfs2 locks the non-existing file. The answer (Created) tells a new empty file has been created. This would be ok if only it were true.

Request:
HEAD /Groups/Arts_and_Science/Curriculum/Committee%20Members%20Homework/Sub-Committee%20A/BIOL/ThisIsATest
Response:
HTTP/1.1 405 Method Not Allowed
Well, this is the end. davfs2 needs this request to get metadata of the newly created file. But suddenly the server no longer allows HEAD, contrary to prior promises. But maybe the file does not exist though it reported 'Created". Finally it is a MUST level requirement in HTTP to include an allow header in a 405 response which isn't there.

Pleas report these bugs to the server's administrator.

I assume (your really should report this clearly) that you had option 'if_match_bug 1' enabled when you created this log file.

You might send another log file without this option.

You might also try the combination of option 'if_match_bug 1' and 'precheck 0'. This will avoid some of the critical HEAD requests.

If you send another log file, please

  • use option 'debug most'
  • use a simple command line tool to create one small file (cp or echo is preferrred over touch. There should be some content in the file)
  • don't do all that ls. It will increase the log file without additional information and makes it harder to analyze.


And never remove irrelevant lines. Whoever is able to decide what is relevant will also be able to do the rest. No need to ask me. (You may replace confidential information by XXX)

Werner

- <_71007>
Mon 14 Nov 2011 12:46:32 PM UTC, comment #4: 

The file ThisIsATest was created by doing

touch ThisIsATest

in a directory to which I should be allowed to upload.  The only other operations were several ls calls to watch it till it disappeared.

Bob Tennent <rdtennent>
Mon 14 Nov 2011 12:39:55 PM UTC, comment #3: 

I edited /etc/rsyslog.conf to log debug messages and obtained the attached (with irrelevant lines edited out).

(file #24369)

Bob Tennent <rdtennent>
Mon 14 Nov 2011 03:49:15 AM UTC, comment #2: 

I tried if_match_bug 1 in both

/etc/davfs2/davfs2.conf

and

/usr/share/davfs2/davfs2.conf

with no change.  I then added

debug most

in both, and can't find debug messages anywhere in /var/log.

Bob T.

Bob Tennent <rdtennent>
Sun 13 Nov 2011 08:17:42 PM UTC, comment #1: 

Most probably this is what happens:
When you copy a file into your davfs2 file system davfs2 will first request a lock for this filename from the server. The server creates an empty file and locks it (you can see the file on the server). When davfs2 tries to upload the file the upload fails, davfs2 frees the lock and the server removes the empty file.

There are many reasons why uploading the file may file. Most popular is a bug in old versions of Apache. (I don't know about Xythos. Maybe it builds on Apache maybe not.) You can try option 'if_match_bug 1' in your davfs2 configuration file.

If this does not help you should instead add option 'debug most'. This will cause a lot of debug messages from mount.davfs in one of your log files. Send them so I can see what is going on.

lost+found:
Every davfs2 file system includes a directory named lost+found. If davfs2 can't upload a file to the server and it looks like this will not change in the future davfs2 will move your file into this directory, with some arbitrary characters appended to the file name.

Werner

- <_71007>
Fri 11 Nov 2011 02:14:35 PM UTC, original submission:  

I've installed davfs2-1.4.6 on a Scientific Linux 6.1 system. I can authenticate to the server, mount a file system locally and download files.  If I upload a file, an ls listing shows the file on the remote system for about two seconds; then the item disappears.  There are no error messages and if there's a lost+found copy of the file somewhere, I haven't found it. I'm not sure which debug feature to turn on.  I've successfully uploaded files to the server using the web interface so presumably the problem isn't one of permissions.  Thanks for any help you can provide.

Bob T.

Bob Tennent <rdtennent>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #24374:  debug.0 added by rdtennent (53KiB - application/octet-stream)
file #24375:  debug.1 added by rdtennent (55KiB - application/octet-stream)
file #24376:  debug.2 added by rdtennent (52KiB - application/octet-stream)
file #24369:  debug added by rdtennent (78KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jlong (Posted a comment)
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by rdtennent (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-07-19 _71007 Open/ClosedOpen Closed
    2011-12-28 _71007 StatusIn Progress Done
    2011-11-15 rdtennent Attached File- Added debug.0, #24374
        Attached File- Added debug.1, #24375
        Attached File- Added debug.2, #24376
    2011-11-14 rdtennent Attached File- Added debug, #24369
    2011-11-13 _71007 StatusNone In Progress
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-caa5.
    Corresponding source code