maindavfs2 - Support: sr #107510, 404 not found error but file...

 
 

sr #107510: 404 not found error but file exists on WebDAV server

Submitter:  Hans Schulz <hschulz>
Submitted:  Sun 07 Nov 2010 10:25:06 AM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 30 Dec 2010 12:01:10 PM UTC, comment #7: 

Hi Werner,
404 not found: i can confirm that your solution works! Thank you very much!

Locking: Yes, I already found that one out, including the irrational behavior of the server as you reported. Setting use_locks to 0 solved that one indeed.

Keep up the good work, much appreciated!
cheers
hans

Hans Schulz <hschulz>
Wed 29 Dec 2010 08:36:37 PM UTC, comment #6: 

Hello Hans,

seems the problem is solved. There are two issues which can be worked around with davfs2's configuration options:

404 Not Found error on unknown file extensions:
Please set option 'add_header Translate f' in your davfs2.conf file. This Microsoft extension is intended to force the server to send the source file and not the output of some server side script (eg. for .asp, .php or .jsp files). But it seems to circumvent the MIME-Type checking of the server alltogether (this is not documented).

Locking problems:
When I tried to use locks I got mysterious 502 Bad Gateway errors. Please set option 'use_locks 0' to avoid this. I did not get this problem with other versions of IIS. Maybe this is due to the fact that Microsoft created some non standard extensions to locking.

Cheers
Werner

- <_71007>
Mon 13 Dec 2010 07:14:41 PM UTC, comment #5: 

Yes, right, the term "loading" in my previous post pertains to downloading the .gpg files. (Uploading poses no problems at all.)

Your suggestion as to try to make the provider modify his server's settings was also the first thing that occurred to me, but I received a flat "no" on my humble and polite request... (honestly, I had not really expected differently).

The server is at https://www.extra-hardeschijf-online.nl hence it is encrypted... I'm afraid I am unable to figure out myself what you suggested since my knowledge on secure networking, setting up dummy servers and clients and the like.I have no clue what a User-Agent-header is to start with. :-)

If I can be helpful by giving you the access credetials to the particular server, I shall gladly do so (in a person to person e-mail of course). Also I will have to collect the information you asked for (see below).

Just to avoid confusion: I mentioned Webconnect, but I should have said that I used Windows Explorer to connect to a web resource (i.e. the Webdav server).
I used Windows XP Professional, version 2002, SP 3 to see if Windows would download the .gpg files. Other info follows via p2p e-mail if that is OK with you.
Thanks!
cheers,
hans schulz

Hans Schulz <hschulz>
Sun 12 Dec 2010 09:18:09 AM UTC, comment #4: 

Please avoid confusion about the word "load". Is it upload (from client to server) or download (from server to client)? Uploading works with davfs2, but downloading fails with 404 due to MIME-type restrictions in IIS.
I assume you are talking about download of .gpg-files.

There is some chance that Microsoft uses non-standard headers for communication between Microsoft clients and servers. But they might just as well use the User-Agent-header for this discrimination. To implement it in davfs2 we need to know what they are doing.

Simple case:
If the traffic is not SSL/TLS-encrypted you can use Wireshark (a Windows version is available) to catch the HTTP-traffic. Please catch all traffic while doing
- connect (create the drive)
- create a file with gpg-extension
- download the file (open it for reading)
It will be lengthy, but send it to me nevertheless.

More difficult:
If they use https it will take some serious effort to back engineer. You will need to create a fake SSL-client as well as a fake SSL-server to get Microsofts requests and replies step by step. GnuTLS and OpenSSL provide tools for this.
If you don't want to do it yourself you need
- patience (I currently have not much spare time)
- send me exact instructions how to connect using "Webconnect", including version numbers of involved programs and service packs installed on Windows XP (some years ago I tried to use Microsoft's mini-redirector and usually failed).

The best solution would of course be to tell your service provider that you have no use for Windows-only services and change the provider. I know it is not that easy with most providers beeing Microsoft-centric (but a lot of them at least use Apache). But you should consider it.

Cheers
Werner

- <_71007>
Sat 11 Dec 2010 11:19:35 AM UTC, comment #3: 

Update: I submitted a complaint at the service provider (KPN), requesting to add .gpg to their IIS metadata so at least I could continue to use duplicity, but the only answer I received was that my "operating system is not supported". Now the funny thing is that using a Windows XP client the 404 Not Found occurs when using Internet Explorer just the same, but remarkably, using a Webconnect to the Webdav server from the Windows Explorer on the client, the .gpg files load without problems. This leads me to suspect that Windows Explorer has a way to tell IIS that the requested MIME type (.gpg) must not be checked at the server (and subsequently blocked with a 404) but to allow loading them instead.
If such a mechanism exists, then could this also be added within dav2fs to allow .gpg (and other mime type) to load correctly?

If it would be possible to monitor the traffic between de XP client and the Webdav server perhaps it can be figured out how the client persuades the server to serve the .gpg files..

Hans Schulz <hschulz>
Thu 11 Nov 2010 08:41:00 PM UTC, comment #2: 

Hello Werner,
I checked as you suggested and found that a file with extension .mp3
loaded without problems, but the duplicity volumes and signature files
(extension .gpg) did not, and neither files without any extension.
Since the server is managed by KPN I have little hope they are willing
(and able) to change things but nevertheless I shall try!
Thanks very much, I will keep you informed!
best regards,
hans

Hans Schulz <hschulz>
Sun 07 Nov 2010 08:53:40 PM UTC, comment #1: 

"I can do "ls -l" on the share and all files are shown.
 ...
 I cannot copy from the share to a local file."
The server is IIS6.

This looks like a known feature of IIS. It only serves files with registered MIME-type and the MIME-type is evaluated from the file extension. So
- you can upload any kind of file with arbitrary file extension
- when you wont to get the file, IIS looks up the file extension in some registry and if it does not find this file extension it will return 404 Not Found.
Please see http://support.microsoft.com/kb/326965 for how to change this behaviour (on the server side).

You should test whether this really is the reason for the problems:
- copy some short files with extensions commonly known on windows systems into the share (e.g. .txt .doc .html .htm)
- copy some short files with extensions probably unknown to windows into the share (e.g. .sh .odt .abc .xyz no extension at all)
- wait at least 20 seconds (upload is delayed in davfs2)
- use ls to see of all the files are there
- copy the files from the share onto your local disk
If you succeed with some extensions (known to windows) but fail with others this IIS feature is the cause of the problem.
The only way to solve it is to configure the server accordingly.

Please tell me the result of your tests.

Cheers
Werner

- <_71007>
Sun 07 Nov 2010 10:25:06 AM UTC, original submission:  

Some background:
I use the (Dutch) KPN service https://www.extra-hardeschijf-online.nl which is a webDAV server, for backup purposes. My backup system is based on duplicity. Since KPN assigns usernames that includes a @ and . character, the standard webdav backend of duplicity gets confused. So I use davfs2 in conjunction with the file:// backend of duplicity.
Status:
making backups to the webdav server works fine. crontab starts a bash script that mounts the webdav share (as root) and duplicity writes the volumes and metafiles to the server neatly. Finally the script unmounts the webdav server from the filesystem, flushing the cache. So far so good.
Problem:
When i try to do "duplicity collection-status" or any other action that requires reading the meta data, such a signature files on the webdav share, i get the error "no such file or directory". This blocks reading back the backups (and that's the idea of backup: being able to restore, isn't it?). Also I cannot do incremental backups.
Diagnostics:
I have mount the webdav share with file_mode and dir_mode set to 755, this makes no difference.
I can do "ls -l" on the share and all files are shown.
I can delete files from the share.
I cannot copy from the share to a local file.
Locks are turned off in my davfs.conf file.
I experimented with read timeouts to no avail.
Cadaver shows the same information: when i read a file it reports a "404 not found" (e.g on cat), "showlocks" in cadaver reports that no locks are owned. File permissions on the share look OK (either 644 or 755).
Since cadaver and davfs2 directly from the terminal all show the same symptoms, I do not suspect duplicity.
Any clues?


Hans Schulz <hschulz>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by hschulz (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-07-19 _71007 Open/ClosedOpen Closed
    2010-12-29 _71007 StatusNeed Info Done
    2010-12-12 _71007 Severity5 - Blocker 3 - Normal
        StatusNone Need Info
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code