taskMapOSMatic - Tasks: task #10040, Gracefuly handle maposmaticd job...

 
 

task #10040: Gracefuly handle maposmaticd job timeouts

Submitter:  David Decotigny <daviddecotigny>
Submitted:  Tue 05 Jan 2010 08:52:47 AM UTC
   
 
Should Start On:  Tue 05 Jan 2010 12:00:00 AM UTC Should be Finished on:  Tue 05 Jan 2010 12:00:00 AM UTC
Category:  daemon Priority:  * 5 - Normal
Status:  Done Privacy:  Public
Assigned to:  daviddecotigny Percent Complete:  0%
Open/Closed:  Closed Release:  None
Effort:  0.00
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 08 Jan 2010 09:44:48 AM UTC, comment #5: 

Suspected domino effect with statement timeout enable: 6988 and next.

David Decotigny <daviddecotigny>
Group administrator
Thu 07 Jan 2010 08:36:28 AM UTC, comment #4: 

For our records, example domino effects: 6070, 6037

David Decotigny <daviddecotigny>
Group administrator
Thu 07 Jan 2010 08:07:28 AM UTC, comment #3: 

Hard coded fix sent to the dev ml and deployed manually to dev+prod servers. The patch configures a query timeout (18mn), which should fire within the allocated time for each rendering (20mn), considering that the problematic query appears to be the 1st ones. This should prevent (or at least limit) any domino effect.

Marking this item "ready for test" now, but should be considered "almost closed".

David Decotigny <daviddecotigny>
Group administrator
Tue 05 Jan 2010 09:57:59 AM UTC, comment #2: 

Nothing tells us that the timeout on layer 'leisure' is due to previous timeouts.

There are many other timeouts in the logs, and the following renderings do not necessarly fail.

Moreover, before trying to make the request asynchronous, maybe we should try to see if the request could be optimized (better indexes ? better request ?)

Thomas Petazzoni <tpetazzoni>
Group administrator
Tue 05 Jan 2010 09:40:52 AM UTC, comment #1: 

There is something somewhat more frightening: after a rendering timed out, the next one gives this: timeout expired in layer 'leisure' (job 5205+6 on prod). This is a bit of a problem... for example if the latter error happens inside the osm2pgsql daily update because of a failed rendering, then I don't know how osm2pgsql would react.

David Decotigny <daviddecotigny>
Group administrator
Tue 05 Jan 2010 08:52:47 AM UTC, original submission:  

We experienced a general failure of the rendering after one ocitysmap job (from maposmaticd) timed out. The failed job appeared to have been stuck (and killed) in the middle of the big street_index queries, and left the corresponding postgresql worker process eating 100% CPU.

This had the exceptional benefit of preventing any other rendering coming afterwards to time out too. The most annoying feature of this behavior being that, if a daily planet update appears, then it will probably take a veeeeery long time to complete. Note that it could be too dangerous to kill the faulty pgsql worker process eating the CPU while the update has already been started (because in that case, I bet nothing would guarantee the consistency of the DB, so we'd better restart the whole pgsql server and, hence, the daily update; but I'm afraid the update process is not idempotent).

One solution we could try: instead of executing the big street_index queries synchronously, run them asynchronously and attach a process atexit/int/quit/term signal handler that calls pg_cancel_query. This /might/ (?) help... maybe.

David Decotigny <daviddecotigny>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dmentre (Updated the item)
  • -email is unavailable- added by tpetazzoni (Posted a comment)
  • -email is unavailable- added by daviddecotigny (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-01-21 daviddecotigny StatusReady For Test Done
        Open/ClosedOpen Closed
    2010-01-12 dmentre Assigned toNone daviddecotigny
    2010-01-07 daviddecotigny Priority8 5 - Normal
        StatusNone Ready For Test

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code