omarshehata / lda-explorable Goto Github PK

It was suggested on /r/statistics that thinking of LDA as an optimization problem as described in this article isn't how statisticians currently think about LDA. This would explain why it is not actually solved as an optimization problem in practice (which is something I skipped over in the article, see #3):

While I applaud the use of interactivity, I don't actually think this is the best way to go about thinking about LDA.

Firstly, you're talking about Fisher's original formulation of LDA (wiki). Nowadays we usually use the generative model version of LDA, and I think that is actually very intuitive.

Essentially, you assume that your data is generated from normal distributions, with a common covariance structure (if it's not the same, then you get QDA). That is, each class has its own normal distribution. Then, it's a little work to show that if you assume those distributions, then the (Bayes) optimal way to classify new data points correspond to linear separations (intuitively, you're just checking which density is higher, that is your classification).

I think reformulating this explanation would essentially be a different article, but it could still re-use most of the code and visualization here. Happy to support anyone who wants to explore this path.

Th article says:

This particular metric is what Fisher chose, but it's certainly not the only possible one. A different metric could optimize for different results (perhaps you care a little more about minimizing scatter so you multiply the denominator by a large number to give it more weight).

However, if you just multiply the denominator by a constant, it changes all the scores the same way, so it won't change the relative "best". See:

https://twitter.com/D_M_Gregory/status/1069965514120851457

Maybe come up with a better example?

The article never explains why Fisher's formula has a denominator of (S1 + S2) as opposed to, say (S1 * S2). This is something I haven't been able to figure out, but some more explanation of how to arrive at that as opposed to other forms would be helpful.

Something I completely skipped over is how to actually solve this optimization problem. Partially because I thought it was outside the scope, but also partially because I don't know it well enough myself to explain it simply. It would be nice to say something about whether it's an iterative solution, whether it's some sort of gradient descent, or whether there's a closed form solution.