bugdavfs2 - Bugs: bug #41355, Partial compression support for...

 
 

bug #41355: Partial compression support for davfs2

Submitter:  Evili del Rio <evili>
Submitted:  Sun 26 Jan 2014 06:45:09 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  _71007
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 12 Mar 2014 04:57:17 PM UTC, comment #11: 

Fixed in CVS branch MAIN and branch rel-1-4-5.

- <_71007>
Sat 01 Feb 2014 04:50:16 PM UTC, comment #10: 

It is not a read-to-use open source file server (compared to most WebDAV uses), but a closed-source business application. It is based on GlassFish v3 plus "WebDAV Support for JAX-RS" (= our project) library and some Servlet filters. But if you like, we can provide tests results once neon+davfs2 adds TE support. :-)

Markus KARG <mkarg>
Tue 28 Jan 2014 07:28:06 PM UTC, comment #9: 

Good to hear about a WebDAV-server that supports TE-compression.

Though have no time to care about this in the near future and though it should probably go into the Neon-library could you please give some more information about this WebDAV-server for future use.

Werner

- <_71007>
Mon 27 Jan 2014 06:00:50 PM UTC, comment #8: 

I'd like to annotate that our WebDAV server does support TE and I would be happy if one day davfs2 would provide support for this. Support for Content Encoding is in fact the wrong way to go, as Werner already pointed out. :-)

Markus KARG <mkarg>
Sun 26 Jan 2014 06:45:09 PM UTC, comment #7: 

This item has been reassigned from the project davfs2 support tracker to your tracker.

The original report is still available at support #108431

Following are the information included in the original report:

[field #0]                  Item ID: 108431
[field #1]                 Group ID: 10199
[field #2]              Open/Closed: Open
[field #3]                 Severity: 3 - Normal
[field #4]                  Privacy: Public
[field #8]                         : Unknown support Field Display Type
[field #9]                 Category: None
[field #10]             Submitted by: evili
[field #11]              Assigned to: wbaumann
[field #12]             Submitted on: Di 05 Nov 2013 09:49:28 GMT
[field #13]                  Summary: Partial compression support for davfs2
[field #14]      Original Submission:
Hello,

I wanted to have support for compression in davfs so I have been playing with the code. I have arrived to a solution that works without changing a lot of code (just a configuration option, some constant definitions, two new functions, and a modification on a data structure. All these modifications are only compiled in if there is support for zlib at compile time.
This modification only works for reading (aka downloads), not for sending new contens (upload), I will try to implement both in the future.

FYI, I pass you a patch against the 1.4.7 version, feel free to incorporate if you wish with the same license as davfs2.

To apply this patch:

% wget https://raw.github.com/evili/davfs2-zlib/master/davfs-zlib.patch
% cd davfs-1.4.7
% patch -p1 < davfs-zlib.patch
% ./bootstrap

Cheers,

 Evili

[field #16]               Item Group: None
[field #17]                   Status: In Progress
[field #18]        Component Version: None
[field #19]         Operating System: GNU/Linux
[field #20]          Reproducibility: None
[field #21]               Size (loc): None
[field #22]            Fixed Release: None
[field #23]          Planned Release: None
[field #24]                   Effort: 0.00
[field #28]                 Priority: 5 - Normal
[field #31]         Percent Complete: 0%
[field #33]                  Release: None
[field #58]     Custom Select Box #1: None
[field #59]     Custom Select Box #2: None
[field #60]     Custom Select Box #3: None
[field #61]     Custom Select Box #4: None
[field #62]     Custom Select Box #5: None
[field #63]     Custom Select Box #6: None
[field #64]     Custom Select Box #7: None
[field #65]     Custom Select Box #8: None
[field #66]     Custom Select Box #9: None
[field #67]    Custom Select Box #10: None

- <_71007>
Wed 13 Nov 2013 08:25:05 AM UTC, comment #6: 


Hello Werner,

These are great news for me! Thanks for accepting my patch.

Other servers: If I don't remember wrong, I tested some servers/configurations (the ones available in Linux) with litmus (http://www.webdav.org/neon/litmus/) and the only that adjusted well for our servers was Apache. Anyway, I will probably retest some of them with simpler configurations and I will let you know.

Cheers,

  Evili

Evili del Rio <evili>
Sat 09 Nov 2013 08:22:18 PM UTC, comment #5: 

Hello Evili,

you finally convinced me: your patch is working as it should. So I will give it a try, test it and probably include it in davfs2. I will probably make it an option configurable in davfs2.conf. Wether it will be default on or off will depend on wether there will be problems with important servers or not.

I am still not happy with what server and browser vendors are doing here. Its protocol abuse. The coppression is intended for transfer only and not part of the resource. But they use content-encoding instead of transfer-encoding. More important: it really violates the spec and e.g. the Apache developers know of this.

The violation of the spec is this: with header Content-encoding: gzip the entity is a different version of the resource from that without compression. They change Content-lenght. They also MUST change Etag. Etag is used e.g. for partial transfer which requires that not a single byte is changed when the Etag is the same. Some peaple argue that this will cause problems with proxies (I don't know whether that really happens).

On the other sid, for use with davfs2 it is important that they do not change the Etag. So we rely on non-complient behaviour.

But this protocol violation seems to be established as a quasi standard, so I agree with using it for davfs2 (but not happy).

If you do, or have done, tests with other servers than Apache, please let me know.

Cheers
Werner

- <_71007>
Fri 08 Nov 2013 10:21:14 AM UTC, comment #4: 


Hi Werner,

Ok, I won't discuss further about RFC 2616 and such. But let me say, at least, that Accept-Encoding/Content-Encoding is the common practice and de facto standard:

http://en.m.wikipedia.org/wiki/HTTP_compression

In any case, I will answer your practical questions about my patch as best as I can:

"Please try: Copy compressed files (different compression formats and different file extensions) to your davfs2 file system...":

I have already made many tests of this kind (I wouldn't have dared to submit you a patch otherwise!). I will put you one example of my tests:

# mount via davfs2 (zlib patched).
% mount https://myserver.dom.ain/userdir /home/localuser/Dav/userdir

# make a local copy to a "disk" filesystem (so that every file is "GET" from server).
% cp -apvi /home/localser/Dav/userdir /var/tmp/userdir-davfs2

# make a direct copy from the server (using rsync, scp, choose your weapons...)
% rsync -auv user@myserver.dom.ain:/real/path/to/userdir/.  /var/tmp/userdir-rsync

# tell me any difference excluding .DAV files (generated by the DAV-server itself to keep DAV information in it)
#      and the lost+found dir generated by the local system.
% diff --recursive  --exclude=lost+found  --exclude=.DAV /var/tmp/userdir-davfs2 /var/tmp/userdir-rsync
# nothig is reported, the copy is EXACT, byte by byte, as it MUST be.

The content of "userdir" is a mix of files, text, images, and compressed content:

% ls -1s /home/localser/Dav/userdir

 95 aa2.txt
571 AdministrationGuidePatchPro2.2.pdf
  3 autoexpect.gz
  1 lost+found
  1 mod_userinfo
 14 start-here.png

And I can show you the log of the server where it shows how some files are transferred:

#Format:
# "REQUEST" RESPONSE-CODE RESPONSE-LENGTH [ORIGINAL-LENGTH/COMPRESSED-LENGTH (COMPRESSION LEVEL)]

"GET /userdir/aa2.txt HTTP/1.1" 200 11539 [96764/11521 (11%)]
"GET /userdir/autoexpect.gz HTTP/1.1" 200 2718 [-/- (-%)]
"GET /userdir/start-here.png HTTP/1.1" 200 13938 [-/- (-%)]
"GET /userdir/AdministrationGuidePatchPro2.2.pdf HTTP/1.1" 200 480404 [584206/480386 (82%)]

So aa2.txt (a plain text file) has been transferred compressed saving almost 90% of network use; autoexpect.gz and start-here.png are transferred without compressing; and AdministrationGuidePatchPro2.2.pdf is compressed (saving only a 20%). mod_userinfo is a directory (with a bunch of files of serveral types).

And remember, independently of the original content and of the way they are transferred, the result is an exact copy of the original files.

"What are the server side rules to decide whether to send "Content-Encoding: gzip" or not?"

It doesn't matter to the client: if the client asks for compression (Accept-Encoding) the sever CAN send the response body compressed (sending a Content-Encoding header). Anyway, the client MUST decompress if (AND ONLY IF) the server tells it to do so. The client shouldn't try to guess the content-type, just follow the protocol.

A really bad configured server could compress-encode everything it sends to a client that asks for compress-encoding: it would be a waste of time, bandwith & CPU, but protocol-compliant.

Usually, a real world sever is normally configured in either of two ways:

a) ONLY compress some kind of content (text, html, javascript, etc).

b) COMPRESS anything except some kind of content: (images, videos, music, gzip, zip, bzip2, ...)

In any case, a well behaved server only compress-encodes the response when the client asks for it.

You can use wget (or curl, etc) to see how this happens:

a) A client that does not want compression:

% wget --debug - https://myserver.dom.ain/userdir/aa2.txt

---request begin---
GET /userdir/aa2.txt HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: /
Host: chest.iri.upc.edu
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Fri, 08 Nov 2013 09:32:18 GMT
Server: Apache
Last-Modified: Fri, 08 Nov 2013 08:26:03 GMT
ETag: "52a0012-179fc-4eaa622086ff2"
Accept-Ranges: bytes
Content-Length: 96764
Cache-Control: max-age=2592000
Expires: Sun, 08 Dec 2013 09:32:18 GMT
Vary: Accept-Encoding,User-Agent
Keep-Alive: timeout=120, max=9999
Connection: Keep-Alive
Content-Type: text/plain; charset=UTF-8

---response end---

b) A client that asks for compression (Accept-Encoding):

% wget --debug  --header='Accept-Encoding: gzip' https://myserver.dom.ain/userdir/aa2

---request begin---
GET /files/users/cciri01/aa2 HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: /
Host: chest.iri.upc.edu
Connection: Keep-Alive
Accept-Encoding: gzip

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Fri, 08 Nov 2013 09:39:20 GMT
Server: Apache
Last-Modified: Fri, 08 Nov 2013 08:26:03 GMT
ETag: "52a0012-179fc-4eaa622086ff2"
Accept-Ranges: bytes
Cache-Control: max-age=2592000
Expires: Sun, 08 Dec 2013 09:39:20 GMT
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Content-Length: 11539
Keep-Alive: timeout=120, max=9999
Connection: Keep-Alive
Content-Type: text/plain; charset=UTF-8

---response end---

As you can see the server transfers the same original content in two differnt encodings, the only headers that change are the ones related with the request itself: Date, Content-Length, Content-Encoding (of course!), and some others (Vary, Expires,...). The headers related to the original file content remains the same: Last-Modified, ETag, Content-Type.

In conclusion: Accept-Encoding/Content-Encoding works well and CORRECLY for davfs2 with the patch I submitted.

HtH.

Cheers,

   Evili

Evili del Rio <evili>
Wed 06 Nov 2013 09:24:19 PM UTC, comment #3: 

I don't agree with your interpretation of RFC 2616. It is one sentence out of context. So here some more of RFC 2616, 14.11 Content-Encoding:

"The Content-Encoding entity-header field is used as a modifier to the media-type. When present, its value indicates what additional content codings have been applied to the entity-body, and thus what decoding mechanisms must be applied in order to obtain the media-type referenced by the Content-Type header field. Content-Encoding is primarily used to allow a document to be compressed without losing the identity of its underlying media type. ... [example] ... The content-coding is a characteristic of the entity identified by the Request-URI. Typically, the entity-body is stored with this encoding and is only decoded before rendering or analogous usage. However, a non-transparent proxy may modify the content-coding if the new coding is known to be acceptable to the recipient, unless the "no-transform" cache-control directive is present in the message."

The most important sentence is: "The content-coding is a characteristic of the entity identified by the Request-URI." Yes, it is part of the file type (in case of a WebDAV file system). Also not the "only decoded before rendering". This is typical browser usage; a file system does not render.

The transfer encoding headers are mainly used for chunked encoding. But this is not because of the spec. The spec allows any kind of transfer encoding, especially it mentions compression as one kind of transfer encoding. But almost nobody implements it and that's why we can't use it. But it would be the right thing for a WebDAV file system.

Practical question:
What are the server side rules to decide whether to send "Content-Encoding: gzip" or not? I assume most servers will do this according to the file extension. When the file extension is somethin like .gz, .zip etc. they will set "Content-Encoding: gzip".

Please try:
Copy compressed files (different compression formats and different file extensions) to your davfs2 file system.
When you do a GET; will the server set Content-encoding? Will your code decompress the file?

When I store a compressed file in my file system and get back a decompressed version the next time I access the file, then the file system has a bug.

Werner

- <_71007>
Wed 06 Nov 2013 08:46:57 AM UTC, comment #2: 


Hi Werner,

IMHO, Content-Encoding is the correct header(*). Quoting RFC-2616: "Content codings are primarily used to allow a document to be compressed or otherwise usefully transformed without losing the identity of its underlying media type and without loss of information"

The Transfer-Encoding header applies mainly to chunked transfers, not to compression(**).

"The files are compressed for transfer and decompressed by the receiver. This way the user of the file system would get files from the davfs2 file system exactly as they where stored there": Yes, this is exactly what my patch does (but using the correct/accepted/standard header for that: Content-Encoding).

In fact, it is the client who must ask for compression sending an Accept-Encoding header with the values "gzip", "deflate". Then the server can optionally send the request compressed with the Content-Encoding set to "gzip" and/or "deflate".

My patch follows the protocol and only inflates when the response from the server is marked with the C-E header. In fact, the server can ignore the A-E header and send the content as is (useful for already compressed content like images, music, video, zips, gzips, etc.).

"Content-encoding tells something about the file type": Nope. You are confusing "Content-Type" with "Content-Encodig".

In summary, my patch respects the original content in a transparent way as davfs2 needs and it works well with protocol-compliant HTTP servers (like Apache with mod_deflate, for example).

I think it merits a second look at least.

Cheers,

  Evili

---
(*) See RFC-2616 (HTTP 1.1 protocol: http://tools.ietf.org/html/rfc2616
(**) The documentation is a little bit confusing because defines the 'encodings' toghether, both for transfer-encoding and content-encoding.

Evili del Rio <evili>
Tue 05 Nov 2013 08:10:53 PM UTC, comment #1: 

There is a problem with this patch, or more precisely there is a problem with HTTP and the way this protocol is implemented by most servers.
(I did not take the time to reread the HTTP-spec and also did not study your patch in detail. So please correct me if I'm wrong.)

HTTP knows two kinds of compression:
- content-encoding
- transfer-encoding

What WebDAV and a WebDAV file system would need is transfer encoding. The files are compressed for transfer and decompressed by the receiver. This way the user of the file system would get files from the davfs2 file system exactly as they where stored there. Transfer encoding is announced in the TE-header. Almost no server implements transfer encoding. And therefore it would be a mostly useless effort to implement this in davfs2.

Content-encoding tells something about the file type. A compressed file is not the same as a file that contains semantically the same content but without compression. davfs2 does not and should not change anything here. If I store a compressed file in my davfs2 file system, I want to see and get a compressed file the next time I mount this file system. I do not want the file system to change my files. (But I think this is what your patch does?)

This is different with browsers. It is ok for them to decompress a file to display the content. Unfortunately server implementers and browser vendors created confusion here. Compression is used to reduce the traffic. They should use transfer encoding for this, but they mostly use content encoding. So many people don't know the difference. But a file system is different from a browser. It should use encoding to reduce traffic, but it should not change compressed files.

Werner

- <_71007>
Sun 26 Jan 2014 06:45:09 PM UTC, original submission:  


Hello,

I wanted to have support for compression in davfs so I have been playing with the code. I have arrived to a solution that works without changing a lot of code (just a configuration option, some constant definitions, two new functions, and a modification on a data structure. All these modifications are only compiled in if there is support for zlib at compile time.
This modification only works for reading (aka downloads), not for sending new contens (upload), I will try to implement both in the future.

FYI, I pass you a patch against the 1.4.7 version, feel free to incorporate if you wish with the same license as davfs2.

To apply this patch:

% wget https://raw.github.com/evili/davfs2-zlib/master/davfs-zlib.patch
% cd davfs-1.4.7
% patch -p1 < davfs-zlib.patch
% ./bootstrap

Cheers,

 Evili

Evili del Rio <evili>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mkarg (Posted a comment)
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by evili (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-04-22 _71007 Open/ClosedOpen Closed
    2014-03-12 _71007 StatusConfirmed Fixed
    2014-01-26 _71007 StatusNone Confirmed
        Assigned toNone _71007
    2014-01-26 _71007 Reassign itemdavfs2, sr #108431 davfs2, bug #41355

    Back to the top

    Powered by Savane 3.14-1244.
    Corresponding source code