Determine a policy for dealing with non-invertible covariance matrices in MCD. about macrobase HOT 4 CLOSED

My hack was apparently not the biggest hack: "With a singular sample covariance matrix, Mplus automatically does a gentle ridging (adding epsilon to the diagonal)." http://www.statmodel.com/discussion/messages/14/1857.html?1425832063

from macrobase.

The obvious approach here is to perform dimensionality reduction. However, if we're dealing with a small sample of data, we may not want to go back and perform dimensionality reduction on the entire dataset. This is even harder in a streaming setting.

I'm generally finding three major alternatives referenced in discussions and notes:

1. "Ridging" the covariance matrix. Add \alpha*I to the covariance matrix, for some small \alpha.
   • Example: http://www.statmodel.com/discussion/messages/14/1857.html?1425832063
2. Compute a pseudo-inverse. A common suggestion for use in Mahalanobis distance is the Moore-Penrose inverse
   • Example: http://stats.stackexchange.com/a/37782
   • Example: http://www.sciencedirect.com/science/article/pii/0146664X79900522
3. Drop the problematic dimensions and compute covariance/Mahalanobis distance in a subset of dimensions.
   • Example: http://www.mathworks.com/matlabcentral/newsreader/view_thread/235323

There are other, less common measures that don't appear to have been widely used. Here's a survey from someone's class notes: http://webcache.googleusercontent.com/search?q=cache:goB_0E5S-BIJ:www.doc.ic.ac.uk/~dfg/ProbabilisticInference/IDAPILecture17.pdf+&cd=4&hl=en&ct=clnk&gl=us

from macrobase.

Someone wrote about this in a real textbook: https://books.google.com/books?id=uL6-PKdS0lAC&pg=PA271&lpg=PA271&dq=mahalanobis+generalized+inverse

from macrobase.

This unblocks us for now, but I'm happy to revisit the ridge technique, especially if SVD proves a bottleneck.

from macrobase.