bugGNU Astronomy Utilities - Bugs: bug #59904, Large aperture can easily fill...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #59904: Large aperture can easily fill memory in sort-based match

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Mon 18 Jan 2021 01:08:10 PM UTC
   
 
Category:  Match Severity:  3 - Normal
Item Group:  Crash Status:  Confirmed
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Mon 18 Jan 2021 01:11:14 PM UTC, comment #1: 

Small typo: at the end of the original post, the aperture size written in text (0.1 arcsec) didn't match the example option value (0.5 arcsec).

They should both be the same value (any one of 0.1 or 0.5 is fine, with 0.1 giving a faster result).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 18 Jan 2021 01:08:10 PM UTC, original submission:  

The sort-based match algorithm which is currently used in the Match program makes a linked list of nearby points (within the given aperture) between the two catalogs to find the best match between them. This is necessary to make sure that the order of the input catalogs doesn't affect the final result (see the comments in the 'match_coordinates_second_in_first' function for more).

However, this has a bad side-effect: when there are many points in both catalogs (for example x100000) and the aperture is large (by mistake usually!), the created lists for each point in each catalog can easily fill the whole system RAM causing Match to crash!

For example, with these two commands we can download the ID, RA and Dec of the same region (randomly selected) in Gaia DR2 and eDR3 (each is 72Mb, with more than 2 million rows):


astquery gaia --dataset=dr2 -csource_id,ra,dec --center=281.6553922,11.4038964 --radius=2 -odr2.fits
astquery gaia --dataset=edr3 -csource_id,ra,dec --center=281.6553922,11.4038964 --radius=2 -oedr3.fits


When we later try to match these two by RA and Dec with the command below, the RAM consumption will exceed 10GB and cause a crash on many systems!


astmatch dr2.fits edr3.fits --ccol1=ra,dec --ccol2=ra,dec --aperture=1


To fix this problem, we should avoid keeping the more distant elements in the list and only keep the top N nearest elements (for example N=10). We just have to use a structure like Gnuastro's "Ordered list of size_t" structure: 'gal_list_osizet_t'.

Until this problem is fixed, to avoid the problem, you should decrease the aperture size to physically meaningful values (in the case of Gaia, something like 0.1 arcsec ('--aperture=0.5/3600').

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code