I'm not sure if this is strictly a bug, but:
sktrafo_compare in Sketch/Modules/sktrafo.c doesn't provide a <a href="http://www.sgi.com/tech/stl/StrictWeakOrdering.html">strict weak ordering</a>.  E.g. it isn't transitive: we can have three SkTrafo objects a,b,c such that a<b and b<c yet a==c.

This has various consequences.  E.g. sort() on a list of SkTrafo's isn't guaranteed to put the list into order; and a dictionary implemented as a tree won't work for SkTrafo keys.

To make sktrafo_compare provide a strict weak ordering, you could change the comparison to as if it were a sequence of 6 doubles, with m11 being the major sort key and so on.
The simplest implementation would be a bitwise comparison memcmp(&a->m11, &b->m11, 6*sizeof(double)), which is robust against NaN.
If you wanted to use floating point comparisons rather than bitwise comparison, then you'd have to test explicitly for isnan, and have NaN compare, say, greater than all non-NaN values (as Java's Double.compareTo does) and equal to all NaN values.