I've written two seperate solvers (in 2D), one based on Muller's source code ( http://graphics.ethz.ch/~mattmuel/demos ... Source.zip) which uses jacobi rotations to find the eigen-stuff, and another method which finds the eigen-stuff directly (http://www.dgp.toronto.edu/~hertzman/co ... eig2x2.cpp).
My problems are with degenerate situations: when the points are colinear, the matrix Apq looses a rank. The direct method fails in this case. After some adjustments, Muller's solver can handle this case -- the rotation matrix R ends up having some scale, so I'm "fixing" this by renormalizing. How is everyone else dealing with this?
My other problem with rank-deficient Apq is that I'm drawing the bodies by extracting a rigid transform (using center-of-mass and the rotation matrix R calculated by the shape matching solver). When Apq has linearly dependent rows, R is no longer the correct body orientation -- it aligns with the axis along which the particles are colinear. This isn't correct: imagine a shape with localspace particle positions (1,1),(0,0),(-1,-1) -- in this case, the particles are colinear along an axis which is diagonal relative to the localspace axes.. however, R aligns the localspace axes with the colinear axis.
I'm also having issues when the shape "flips" inside-out (this phenomena occurs with mass-spring systems, when one particle moves to the wrong side of the shape, reversing the clockness of the shape). Muller corrects this by negating the second column of Apq, however this isn't a correct solution in all cases; when it works, it works well (the shape is translated to prevent the vert from crossing through), however when it doesn't work, things tend to blow up (the shape is rotated by 180deg or more). My biggest problem with this is that I have no idea what to search for.. "flipping"? Inversion and inside-out also suck as search terms

I'm hoping someone has some suggestions or experience they could share; it's possible that I've missed something obvious.