taskMapOSMatic - Tasks: task #9917, Apply optipng (or pngcrush) on...

 
 

task #9917: Apply optipng (or pngcrush) on generated PNG files

Submitted by:  David MentrĂ© <dmentre>
Submitted on:  Mon 16 Nov 2009 08:56:16 AM UTC  
 
Should Start On: Sun 15 Nov 2009 11:00:00 PM UTCShould be Finished on: Sun 15 Nov 2009 11:00:00 PM UTC
Category: ocitysmapPriority: 5 - Normal
Status: CancelledPrivacy: Public
Percent Complete: 10%Assigned to: Jeroen van Rijn <jvrnix>
Open/Closed: ClosedRelease: None
Effort: 0.00

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 07 Aug 2010 11:50:23 PM UTC, comment #6:

Pngcrush takes too long for the size gain, discarding issue.

Maxime Petazzoni <mpetazzoni>
Project Administrator
Tue 02 Feb 2010 08:42:12 PM UTC, comment #5:

There is that, and taking into consideration that modern browsers (Fx 3.x, Chrome, Safari, and I think Opera too) support SVG people can even use them for previewing the files if they don't have a vector editor like Inkscape or SodiPodi installed.

This does bring up the question of rendering the thumbnails. Have maposmaticd do a conversion pass using libpoppler on the PDF?

I've not been feeling well the past two weeks (very stubborn cold), hence the absence on IRC as well, but now that that's clearing up I'd be happy to look into any directions you think sensible to explore regarding this task.

Jeroen van Rijn <jvrnix>
Project MemberIn charge of this item.
Tue 02 Feb 2010 08:32:59 PM UTC, comment #4:

Thanks Jeroen for your detailed report. As you say, we need to test on a few more maps (larger maps, smaller maps, full maps, empty maps) and indexes before deciding which parameters give the best result. Note that I'm not particularly impressed by a ~15% reduction in the size, so I'm not sure whether it's really worth the effort.

Concerning the fact that PNG is only a debugging format for Cairo, my proposal is to output only SVG and PDF using Cairo, and then turn either the SVG or the PDF into a PNG file using an external tool (inkscape if SVG is choosen, another tool for the PDF). This would also probably help us solving the fact that the Cairo PNG output is very memory-consuming and is currently the limiting factor for rendering large regions.

Thomas Petazzoni <tpetazzoni>
Project Administrator
Fri 22 Jan 2010 04:49:20 PM UTC, comment #3:

Make sure that if/when geo-data is added to a png, per task #10116, the optimization of the file doesn't scrub it.

Jeroen van Rijn <jvrnix>
Project MemberIn charge of this item.
Sun 17 Jan 2010 11:15:38 AM UTC, comment #2:

I've posed a question to the Cairo list to see if write_to_png could be called with parameters to pass directly to libpng and its use of zlib.

http://lists.nongnu.org/archive/html/maposmatic-dev/2010-01/msg00463.html

Unfortunately it appears that Cairo's PNG output is a debug path never meant for production use; I was advised to see about overloading write_to_png and using a dedicated graphics library to do the actual png writing, which then might also give more flexibility in telling libpng what to do.

My proposal is that I first pursue optimal parameters for optipng as intended and add this support to maposmaticd, then instead of marking this task as closed, direct the effort to finding a suitable python image library which if present (as an optional dependency) allows to write png's (and possible other raster graphics) from Cairo's image context while at the same time directly allowing us to set libpng/zlib parameters for better compression.

Jeroen van Rijn <jvrnix>
Project MemberIn charge of this item.
Sat 16 Jan 2010 07:52:37 PM UTC, comment #1:

I've done some preliminary testing on a limited set of maps, doing the following:

Listed times are on a 5337x3316 pixels, 3x8 bits/pixel map of Haarlem. Recorded on a Phenom II quad core @ 3.2Ghz with 8Gb RAM. Keep in mind that optipng doesn't use multithreading and uses only a single core.

OptiPNG 0.6.2.1: Advanced PNG optimizer. (version that comes with Ubuntu Karmic).

# exhaustive search
time optipng -zc1-9 -zm1-9 -zs0-3 -f0-5 <png>
15 minutes, 23 seconds (real)
# 240 trials - too long to run on the server
time optipng -o7 <png>
4 minutes, 5 seconds (real)

# 16 trials - doable
time optipng -o3 <png>
39 seconds (real)

# about 8 or so trials - much better
time optipng -zm 9 -zc 9 <png>
22 seconds (real)

-o7 came up with a setting that reduced the size the most:
Selecting parameters:
zc = 9 zm = 9 zs = 0 f = 0 IDAT size = 2485922
Output file size = 2485997 bytes (429704 bytes = 14.74% decrease)

I noticed that -o3 covers this range as well, and so does the explicit selection of -zm 9 and -zc 9. They just do so much more quickly.

Conclusions at this point:

  • optipng -o3 is a good tradeoff between time to run and size reduction, particularly because it also found the best parameters.

Todo:

  • Create a harness to create two dozen maps of various parts in the world and their indexes;
    • Have the harness time optipng at exhaustive, -o7, -o3, and zc/zm and use -log;
    • See if the best parameters selected by the exhaustive search for each of the 24 maps are covered by the other, faster searches as well;
    • Select the parameters with the best size/time tradeoff;
    • Run the entire test set 3 times to have more accurate timing
  • Build a system optimised binary and see if it runs significantly faster than the stock Karmic supplied .deb
  • Allow maposmaticd to call optipng with the parameters we find and decide on (read from the configuration file)
    • If it turns out maps compress better with 'x' and indexes better with 'y' and this has no time tradeoff, add a configuration value for both separately.
Jeroen van Rijn <jvrnix>
Project MemberIn charge of this item.
Mon 16 Nov 2009 08:56:16 AM UTC, original submission:

Suggestion by Alain-Pierre Perrin: Apply optipng (or pngcrush) on generated PNG files. It can reduce map file by 10% and index file by 50%.

http://optipng.sourceforge.net/

http://pmt.sourceforge.net/pngcrush/

David Mentré <dmentre>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Digest:
   bug dependencies, task dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mpetazzoni (Posted a comment)
  • -unavailable- added by tpetazzoni (Posted a comment)
  • -unavailable- added by jvrnix (Updated the item)
  • -unavailable- added by dmentre (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 07 Aug 2010 11:50:23 PM UTCmpetazzoniStatusNone=>Cancelled
      Open/ClosedOpen=>Closed
    Fri 22 Jan 2010 04:46:33 PM UTCjvrnixDependencies-=>Depends on task #10116
    Sat 16 Jan 2010 07:52:37 PM UTCjvrnixPercent Complete0%=>10%
    Wed 13 Jan 2010 07:39:29 PM UTCjvrnixDependencies-=>Depends on bugs #27467
    Wed 13 Jan 2010 07:34:23 PM UTCjvrnixAssigned toNone=>jvrnix

    Back to the top


    Powered by Savane 3.1-cleanup