bugDatabase-driven manual pager suite - Bugs: bug #28990, missing native lzma decompression

 
 

bug #28990: missing native lzma decompression

Submitted by:  Jérôme Bouat <jerome_bouat>
Submitted on:  Tue 23 Feb 2010 10:53:45 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Wont Fix
Privacy: PublicAssigned to: None
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 23 Feb 2010 11:51:27 AM UTC, comment #3:

> > Manual pages are typically so small
> > that the compression method
> > you use really doesn't matter much;
>
> Size isn't the only benefit: it decompresser faster.


http://tukaani.org/lzma/benchmarks.html, on the upstream author's own web site, shows substantially slower decompression using lzma than using gzip. In case you object that this was with LZMA1, http://fedoratux.blogspot.com/2009/05/xz-compression.html is quite recent and supports this with XZ.

But, even if it were faster, it would be of fractional benefit at most. Manual pages are not big enough to exhibit substantial benefit from decompression time.

> > compressing them further usually means
> > that they still take up the same number of blocks
> > and thus the same effective disk space.
>
> Even if the same number of blocks are used,
> the decompression will process less data flow.
> Moreover lzma decompresses faster than gzip.


See above. Also, lzma uses more memory to decompress; again, it's probably fractional at the file sizes we're talking about, but it's honest to mention it given that man is a program often used even on low-memory systems.

> > Yes, there is some small benefit,
> > but it's not worth extra complexity
>
> This could be a long term feature only.


I'm not going to lie. If I don't think something should be done, I'm not going to put it on a "long-term wishlist" and then bury it; I'll just say that I don't think it should be done. This is more honest to you, and better for me in terms of managing my bug list.

> > and certainly not worth extra run-time linkage.
>
> You could provide alternative configuration:
> either gzip or lzma (not both).


This has a cost to me as the maintainer, which I don't want to bear, and a cost to users in terms of additional complexity. Since your interest appears to be Ubuntu, this wouldn't benefit you anyway as there's no way I would turn off zlib linkage there; it would not make sense to do so.

For casual use of lzma, using the external program is sufficient, and I consider this an acceptable level of support. I don't think it's appropriate, wise, or necessary for distributions to compress all or most of their manual pages using lzma, and so I don't think it's necessary to have man-db link against liblzma to make it faster. Sorry.

Colin Watson <cjwatson>
Project Administrator
Tue 23 Feb 2010 11:13:55 AM UTC, comment #2:

> Manual pages are typically so small
> that the compression method
> you use really doesn't matter much;


Size isn't the only benefit: it decompresser faster.

> compressing them further usually means
> that they still take up the same number of blocks
> and thus the same effective disk space.


Even if the same number of blocks are used,
the decompression will process less data flow.
Moreover lzma decompresses faster than gzip.

> Yes, there is some small benefit,
> but it's not worth extra complexity


This could be a long term feature only.

> and certainly not worth extra run-time linkage.


You could provide alternative configuration:
either gzip or lzma (not both).

Jérôme Bouat <jerome_bouat>
Tue 23 Feb 2010 11:04:38 AM UTC, comment #1:

I am the upstream maintainer too ...

Sorry, but I really don't think this is worth it, and will not be spending any time on it. Manual pages are typically so small that the compression method you use really doesn't matter much; compressing them further usually means that they still take up the same number of blocks and thus the same effective disk space. Yes, there is some small benefit, but it's not worth extra complexity and certainly not worth extra run-time linkage.

Colin Watson <cjwatson>
Project Administrator
Tue 23 Feb 2010 10:53:45 AM UTC, original submission:

The lzma compression algorithm has the below advantages which perfectly fit the man pages features:
- higher compression ration than gzip
- slightly faster decompression than gzip

Of course, there is 1 drawback: lzma is slow for compressing. However, it should be done only 1 time at package building time.

Below is a comparison of the /usr/share/man/man1/mlocate.1.gz provided by the mlocate package:
-----
j@j-dt:~$ du -b mlocate.1*
6404 mlocate.1
2498 mlocate.1.gz
2461 mlocate.1.lzma
j@j-dt:~$ time { for i in {0..999} ; do unlzma --stdout mlocate.1.lzma >/dev/null ; done }

real 0m15.210s
user 0m4.200s
sys 0m10.930s
j@j-dt:~$ time { for i in {0..999} ; do gunzip --stdout mlocate.1.gz >/dev/null ; done }

real 0m21.759s
user 0m5.200s
sys 0m16.490s
j@j-dt:~$
-----

I posted a bug report for the Ubuntu distro:
https://bugs.launchpad.net/ubuntu/+source/manpages/+bug/445461

The man-db package maintainer told me that man has to fork an external program to decompress lzma files and this could harm the overall performance.

Maybe providing a native lzma support, without an external program, would enhance the global performance of man.

Jérôme Bouat <jerome_bouat>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by cjwatson (Posted a comment)
  • -unavailable- added by jerome_bouat (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 23 Feb 2010 11:04:38 AM UTCcjwatsonStatusNone=>Wont Fix
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1