taskMapOSMatic - Tasks: task #9917, Apply optipng (or pngcrush) on...

 
 

task #9917: Apply optipng (or pngcrush) on generated PNG files

Submitter:  David MentrĂ© <dmentre>
Submitted:  Mon 16 Nov 2009 08:56:16 AM UTC
   
 
Should Start On:  Sun 15 Nov 2009 11:00:00 PM UTC Should be Finished on:  Sun 15 Nov 2009 11:00:00 PM UTC
Category:  ocitysmap Priority:  * 5 - Normal
Status:  Cancelled Privacy:  Public
Assigned to:  jvrnix Percent Complete:  10%
Open/Closed:  Closed Release:  None
Effort:  0.00
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 07 Aug 2010 11:50:23 PM UTC, comment #6: 

Pngcrush takes too long for the size gain, discarding issue.

Maxime Petazzoni <mpetazzoni>
Group administrator
Tue 02 Feb 2010 08:42:12 PM UTC, comment #5: 

There is that, and taking into consideration that modern browsers (Fx 3.x, Chrome, Safari, and I think Opera too) support SVG people can even use them for previewing the files if they don't have a vector editor like Inkscape or SodiPodi installed.

This does bring up the question of rendering the thumbnails. Have maposmaticd do a conversion pass using libpoppler on the PDF?

I've not been feeling well the past two weeks (very stubborn cold), hence the absence on IRC as well, but now that that's clearing up I'd be happy to look into any directions you think sensible to explore regarding this task.

Jeroen van Rijn <jvrnix>
Group Member
Tue 02 Feb 2010 08:32:59 PM UTC, comment #4: 

Thanks Jeroen for your detailed report. As you say, we need to test on a few more maps (larger maps, smaller maps, full maps, empty maps) and indexes before deciding which parameters give the best result. Note that I'm not particularly impressed by a ~15% reduction in the size, so I'm not sure whether it's really worth the effort.

Concerning the fact that PNG is only a debugging format for Cairo, my proposal is to output only SVG and PDF using Cairo, and then turn either the SVG or the PDF into a PNG file using an external tool (inkscape if SVG is choosen, another tool for the PDF). This would also probably help us solving the fact that the Cairo PNG output is very memory-consuming and is currently the limiting factor for rendering large regions.

Thomas Petazzoni <tpetazzoni>
Group administrator
Fri 22 Jan 2010 04:49:20 PM UTC, comment #3: 

Make sure that if/when geo-data is added to a png, per task #10116, the optimization of the file doesn't scrub it.

Jeroen van Rijn <jvrnix>
Group Member
Sun 17 Jan 2010 11:15:38 AM UTC, comment #2: 

I've posed a question to the Cairo list to see if write_to_png could be called with parameters to pass directly to libpng and its use of zlib.

http://lists.nongnu.org/archive/html/maposmatic-dev/2010-01/msg00463.html

Unfortunately it appears that Cairo's PNG output is a debug path never meant for production use; I was advised to see about overloading write_to_png and using a dedicated graphics library to do the actual png writing, which then might also give more flexibility in telling libpng what to do.

My proposal is that I first pursue optimal parameters for optipng as intended and add this support to maposmaticd, then instead of marking this task as closed, direct the effort to finding a suitable python image library which if present (as an optional dependency) allows to write png's (and possible other raster graphics) from Cairo's image context while at the same time directly allowing us to set libpng/zlib parameters for better compression.

Jeroen van Rijn <jvrnix>
Group Member
Sat 16 Jan 2010 07:52:37 PM UTC, comment #1: 

I've done some preliminary testing on a limited set of maps, doing the following:

Listed times are on a 5337x3316 pixels, 3x8 bits/pixel map of Haarlem. Recorded on a Phenom II quad core @ 3.2Ghz with 8Gb RAM.  Keep in mind that optipng doesn't use multithreading and uses only a single core.

OptiPNG 0.6.2.1: Advanced PNG optimizer. (version that comes with Ubuntu Karmic).

# exhaustive search
time optipng -zc1-9 -zm1-9 -zs0-3 -f0-5 <png>
  15 minutes, 23 seconds (real)
# 240 trials - too long to run on the server
time optipng -o7 <png>
  4 minutes, 5 seconds (real)

# 16 trials - doable
time optipng -o3 <png>
  39 seconds (real)

# about 8 or so trials - much better
time optipng -zm 9 -zc 9 <png>
  22 seconds (real)

-o7 came up with a setting that reduced the size the most:
Selecting parameters:
  zc = 9  zm = 9  zs = 0  f = 0 IDAT size = 2485922
Output file size = 2485997 bytes (429704 bytes = 14.74% decrease)

I noticed that -o3 covers this range as well, and so does the explicit selection of -zm 9 and -zc 9. They just do so much more quickly.

Conclusions at this point:

  • optipng -o3 is a good tradeoff between time to run and size reduction, particularly because it also found the best parameters.


Todo:

  • Create a harness to create two dozen maps of various parts in the world and their indexes;
    • Have the harness time optipng at exhaustive, -o7, -o3, and zc/zm and use -log;
    • See if the best parameters selected by the exhaustive search for each of the 24 maps are covered by the other, faster searches as well;
    • Select the parameters with the best size/time tradeoff;
    • Run the entire test set 3 times to have more accurate timing


  • Build a system optimised binary and see if it runs significantly faster than the stock Karmic supplied .deb


  • Allow maposmaticd to call optipng with the parameters we find and decide on (read from the configuration file)
    • If it turns out maps compress better with 'x' and indexes better with 'y' and this has no time tradeoff, add a configuration value for both separately.
Jeroen van Rijn <jvrnix>
Group Member
Mon 16 Nov 2009 08:56:16 AM UTC, original submission:  

Suggestion by Alain-Pierre Perrin: Apply optipng (or pngcrush) on generated PNG files. It can reduce map file by 10% and index file by 50%.

http://optipng.sourceforge.net/

http://pmt.sourceforge.net/pngcrush/

David Mentré <dmentre>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Carbon-Copy List
  • -email is unavailable- added by mpetazzoni (Posted a comment)
  • -email is unavailable- added by tpetazzoni (Posted a comment)
  • -email is unavailable- added by jvrnix (Updated the item)
  • -email is unavailable- added by dmentre (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-08-07 mpetazzoni StatusNone Cancelled
        Open/ClosedOpen Closed
    2010-01-22 jvrnix Dependencies- Depends on task #10116
    2010-01-16 jvrnix Percent Complete0% 10%
    2010-01-13 jvrnix Dependencies- Depends on bugs #27467
    2010-01-13 jvrnix Assigned toNone jvrnix

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code