helpdavfs2 - Support: sr #109020, Files with parenthesis in the...

 
 

sr #109020: Files with parenthesis in the names do not get uploaded to webdav

Submitted by:  Aria <ariabamdad>
Submitted on:  Mon 25 Apr 2016 04:21:45 PM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 3 - NormalStatus: Done
Privacy: PublicAssigned to: Werner Baumann <wbaumann>
Open/Closed: OpenOperating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 16 May 2016 08:22:37 PM UTC, comment #12:

Thanks for your comments Werner. For now, everything is working fine with session cookies turned on from the DAVFS2 side so I guess we can close this ticket. Thanks for your suggestions that prompted me to change this setting and correct the problem.

Aria <ariabamdad>
Mon 16 May 2016 08:38:31 AM UTC, comment #11:

"Is it possible that the code for davfs2 somehow does something differently when it comes to this type of file?"
No, it does not.

  • It treats parentheses according to the specs, like any other client, e.g. Cyberduck.
  • Whether cookies are enabled or not does in no way affect the request line. You can easily check this by looking e.g. at the Apache log file.
  • As you tell davfs2 works easily when connecting directly to Tomcat, even with cookies disabled.

The problem only occurs when the server combines Apache with Tomcat, and even then only when no cookie header is send. It is obvious that it is the server that treats the same request line differently depending on the cookie header. And it only does this when the file name contains parentheses. There is absolutely no logical or technical reason for this. It is a server bug.

I seems that the communication between Apache and Tomcat works differently, depending on the presence of the cookie header. That is the place where I would start searching for the bug.

Probably Tomcat has a log file of its own. Comparing this with the Apache log for requests with parentheses, with and without cookie header, might give a hint.

"The JSESSIONID cookie is needed by Liferay to work properly."
I can't see this. From what I can see JSESSIONID is only needed to not trigger a server bug. If the server creates to many sessions, just disable useless session management on the server side.

"Perhaps the default setting for this option should be changed from 0 to 1 as more and more web applications are using sessions"
Cookies are heavily misused and they are not needed for WebDAV. So, if some service has additional requirements beyond what the spec requires it is up to the provider of this service to inform the users about this. And concerning me: I want to know what for. Just "everybody does" is not a good reason and it is no reason to change the default settings of davfs2. In my opinion a widely misused feature like cookies should only be enabled if explicitely requested by the user and not by default. And with option n_cookies the user can do this.

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Wed 11 May 2016 03:54:29 PM UTC, comment #10:

Hi Werner,

One more update. I can confirm that when I connect to the web application (liferay) directly to Tomcat, I am able to write files with ( or ) in the fine name. However, when I connect via Apache, I cannot.

I am on SLES 11 SP3 and using apache2-2.2.12-59.1 . With Apache, as I mentioned in my last post, I need to enable the session cookies setting for it to work. I also noticed another person open a ticket relating to session variables and excessive number of sessions being opened on the server (webdav) end. Perhaps the default setting for this option should be changed from 0 to 1 as more and more web applications are using sessions (even though webdav does not). Not sure if this would cause unnecessary overhead or not.

The question still remains as to why this only happens with files with special characters and through Apache.

Thanks,
Aria

Aria <ariabamdad>
Wed 11 May 2016 02:34:51 PM UTC, comment #9:

Hi Werner,

I contacted the vendor for the product and their testing showed that they had no problem creating files with special characters when they use davfs2 and communicate directly with tomcat and the application. However, in my environment, I have Apache in between the client and tomcat and Apache uses JK to communicate with tomcat. I was not able to setup an environment where I can connect davfs2 directly to tomcat (long story).

Reading your excellent reply, I began to test using some of the suggestions. I was able to resolve the issue when I added the "allow_cookie 1" option in the configuration file for davfs2. The JSESSIONID cookie is needed by Liferay to work properly. However, this clearly shows that something odd is going on. Because davfs2 is able to successfully create all other files on the server but has trouble with files with special characters like ( or ) and others. Is it possible that the code for davfs2 somehow does something differently when it comes to this type of file? Could Apache have something to do with this? Clearly, other clients don't seem to be bothered, perhaps because they send the cookies with every PUT.

At this point, it seems that we need the 'allow_cookie 1' option to make this work but from what I see, the problem may be in davfs2.

I am happy to run additional testing at my end if you would like. Just let me know.

Thanks for your help.

Aria

Aria <ariabamdad>
Mon 09 May 2016 05:41:03 PM UTC, comment #8:

Thanks Werner. I am working with Liferay support to see if this is a server side issue. Will post back my findings here.

Aria <ariabamdad>
Thu 28 Apr 2016 03:26:11 PM UTC, comment #7:

After reading the log files the behavior of the server looks even more strange than before.
In "cyberduck apache log.txt" lines 1 and 2, as well as lines 4 and 5: the first HEAD request returns status 404 Not Found and the second one, within the same second, returns with 200 OK. All for requests are for the same resource.
Though this is not directly related to the problem in question it indicates that the server might need some fixing.

I could not determine what is going wrong and believe that the problem is a server bug, but there are some possible additional tests.

File name and percent encoding
------------------------------
Percent encoding is a frequent source of confusion and bugs and so it was my first guess. But it isn't.
Comparing line 7 in "cyberduck apache log.txt" with lines 7 and 8 in "davfs apache log.txt" shows that cyberduck and davfs treat parentheses exactly the same (%28 %29).
One difference is the file extension and some servers indeed treat request different depending on the extension (IIS).

What you might test:
Trying to create files with exactly the same name with cyberduck and davfs2. The server log will be sufficient to see what happened.
Names to test would be:
tst(1)
tst
tst(1).pdf
maybe others you can think of.

HTTP error message
------------------
When the server responds with 401 on the PUT request for tst(1), the response contains 954 bytes of message body. This message might give some hint (it might as well be garbage). To view this message in the davfs2 debug log you need to add option "debug httpbody" to your davfs2.conf file (in addition to debug most).

Authentication
--------------
The server returns 401.
But the log file indicates that davfs2 is providing correct authentication data all the time. With the current debug configuration authorization data are hidden. To change this you need "debug http", "debug httpauth" and "debug secrets". The log files will now contain confidential information. Don't send it to this tracker. You can check it yourself:
- with status 401 the server will send a nonce in header WWW-Authenticate.
- the client computes the authorization data from this nonce and the credentials
- the client will use this authorization data for all subsequent request until it gets a new 401 response with a new nonce.
- now the client will compute the new data, and so on.

Cookie
------
The server sends cookie JSESSIONID. By default davfs2 ignores cookies. The server may want to punish davfs2 for this with strange behavior.
You can add option "allow_cookie 1" and davfs2 will send the cookie.

Conditional requests
--------------------
davfs2 by default uses headers If-Match or If-None-Match with PUT. At least one very old version of Apache/mod_dav has a bug regarding these headers.
You may try option "if_match_bug 1" and davfs2 will no longer use these headers.

You may notice that some of my proposals don't look very logical. They are just more or less blind trial and errors, just to be sure before sending a bug report to Liferay. Most probably it is a server bug.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 26 Apr 2016 10:12:37 PM UTC, comment #6:

Werner,

I forgot to answer one of your questions. I don't use an Apache module for Webdav. My apache server passes the requests to Tomcat that is running Liferay portal application. Liferay itself has webdav build in. Here are some links for Liferay:

https://www.liferay.com/
https://en.wikipedia.org/wiki/Liferay
Thanks,
Aria

Aria <ariabamdad>
Tue 26 Apr 2016 08:53:44 PM UTC, comment #5:

Hi Werner,

I enabled debug and mounted the webdav as a directory and then copied a file by the name test(1) to the directory, then waited and unmounted the mount point.

I have attached a zip file with the logs you asked for. I have sanitized the files a bit so please ignore the host names/ip addresses. The two files with names starting with davfs are for the process I mentioned about where I created the test(1) file. The other log file is from yesterday where a PC webdav client deposited a file by the name alla(test_04_25_2016).pdf using Cyberduck.

Let me know if I can get you any other information.

Thanks.

Aria <ariabamdad>
Tue 26 Apr 2016 08:52:07 AM UTC, comment #4:

What might also be useful:

The log file of your webserver will aslo contain entries from the other client that were successful. It would be interesting to see the path string of these entries.

Besides file names with parentheses also file names with other reserved characters would be of interest, like space, &, + etc.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Tue 26 Apr 2016 07:22:40 AM UTC, comment #3:

This sounds real strange. I tested with the WebDAV server of my ISP (Apache with mod_dav) and there is no problem with parentheses. So what we need to do is to watch the HTTP-traffic.

To log the HTTP-traffic please add option "debug most" to your davfs2.conf file.

To keep the amount of log messages small it is important to avoid any uneccessary file system calls. You should
- mount the file system
- create one small file (e.g. 'echo "test" > path/tst\(1\)')
- wait about 20 seconds
- unmount the file system

Which log file contains the debug messages depends on your system. The messages are from mount.davfs.

In addition to these debug messages I would need this information:
- what WebDAV module do you use with Apache? If it is not Apache's own mod_dav a link to some documentation could be useful.
- what other client do you use that does not show the problem? If you are able to fetch the HTTP traffic of this client for comparison this would be great.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon 25 Apr 2016 07:18:44 PM UTC, comment #2:

Hi Werner,

Thanks for your quick reply. Yes, I am sure the user name and password is correct. We use your davfs2 tool extensively (thank you) to synchronize a network drive with webdav. The tool is used 100's of times in a script. We noticed that when the file names have certain special characters, we are unable to upload files so we assumed these characters were not supported by the application. However, we tested this with another webdav client and the file uploaded file, through the same we server (Apache) so I am sure it's not the web server.

I tested this today by mounting a webdav URL as a local directory, then writing a test file with and one without parenthesis in the file name. The one with parenthesis failed and I provided the error messages in my original post. It seems that when the file name has parenthesis, for some reason the connection fails to provide authentication information. Hard to believe. Please let me know what you would like me to test to provide more information but surely, it is not a simple invalid user name or password issue because the file system is mounted correctly and other files are properly uploaded.

Thanks

Aria <ariabamdad>
Mon 25 Apr 2016 07:09:21 PM UTC, comment #1:

Status code 401 means unauthenticated. So you probably did not provide the proper username and password to davfs2.

The path shown by your webserver "/webdav/grade/document_library/Private/Reports/tst%281%29" is correct. It is the percent encoded form of "/webdav/grade/document_library/Private/Reports/tst(1)".

Are you sure you can copy files without paranthesis in the name to your davfs2 file system and they are accepted by the server? If this is the case (I doubt it) then there is something terribly wrong with your webserver configuration.

Werner

Werner Baumann <wbaumann>
Project AdministratorIn charge of this item.
Mon 25 Apr 2016 04:21:45 PM UTC, original submission:

Hi,
I am using davfs2 1.4.7 to mount a webdav file system as a local directory as root. When I create/copy/move a file to the local webdav mounted filesystem that has a parenthesis in the file name, the file is not written to the webdav and ends up in the lost+found directory.

When the file with parenthesis in the name is created, I receive the following error in /var/log/messages:

Apr 25 12:04:10 myserver mount.davfs: authentication failure:
Apr 25 12:04:10 myserver mount.davfs: PortalRealm

At the same time, I get the following in my Apache log:

myserver.my.domain 192.168.111.1 - - [25/Apr/2016:12:04:10 -0400] "PUT /webdav/grade/document_library/Private/Reports/tst%281%29 HTTP/1.1" 401 954 "-" "davfs2/1.4.7 neon/0.29.6"
myserver.my.domain 192.168.111.1 - - [25/Apr/2016:12:04:10 -0400] "PUT /webdav/grade/document_library/Private/Reports/tst%281%29 HTTP/1.1" 401 954 "-" "davfs2/1.4.7 neon/0.29.6"

In the above, a file by the name tst(1) was created.

When I transfer similar files using a webdav client on a PC, it works fine.

Thanks in advance.

Aria <ariabamdad>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #37001:  logs.zip added by ariabamdad (5KiB - application/zip - log files)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by wbaumann (Posted a comment)
  • -unavailable- added by ariabamdad (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 18 May 2016 06:17:21 PM UTCwbaumannStatusIn Progress=>Done
    Tue 26 Apr 2016 08:55:01 PM UTCariabamdadAttached File-=>Added logs.zip, #37001
    Mon 25 Apr 2016 07:09:21 PM UTCwbaumannStatusNone=>In Progress
      Assigned toNone=>wbaumann

    Back to the top


    Powered by Savane 3.1-cleanup1