Correlation function needs reconsideration about pynndescent HOT 6 CLOSED

Two variables being perfectly anti-correlated (r = -1) implies a relationship, but it definitely doesn't imply similarity.

The current implementation is the definition of Pearson's distance.

You could certainly implement 1 - |corr| as a custom metric if you wished, if that makes sense for you.

from pynndescent.

Yes, it does. Both 1 and -1 Pearson correlation coefficient corresponds to cosine similarity of 1(or near 1), at least for the positive space. Because corr coeff is simply a cosine similarity in the sample space centered at the mean.

Yes, I understand the attempt to convert correlation into distance by subtracting it from 1. However, this does not work for correlation coefficient.

Currently I am defining 1-|corr| but I think I will just stick to 1-cos instead.

from pynndescent.

Yes, it does. Both 1 and -1 Pearson correlation coefficient corresponds to cosine similarity of 1(or near 1), at least for the positive space.

??? This just isn't true. Maybe I misunderstand the requirements for what you are describing, but e.g. np.arange(10) and np.arange(10)[::-1] are perfectly anti-correlated and certainly not cosine similarity of 1.

from pynndescent.

Thanks James, I just checked the cor is -1 but the cos is 0.571, but both cor and cos between np.arange(10) and np.arange(10) was 1. So, looks like cos distance does differentiate between positive and negative correlation, but considers positive correlation to be more similar than negative correlation. Both positive and negative correlation are similar (\in [0,1]) wrt cosine.

Anyways, the argument about 1-cor distance still holds. For negative correlation, the distance is large, and when pydescent finds NN its will only consider neighbor with positive cor and neglect neighbor negative cor. However, in reality finding a strong negative correlation is just as important.

from pynndescent.

Anyways, the argument about 1-cor distance still holds. For negative correlation, the distance is large, and when pydescent finds NN its will only consider neighbor with positive cor and neglect neighbor negative cor. However, in reality finding a strong negative correlation is just as important.

This will always be application and data-dependent, though. Sometimes it's important, sometimes it isn't, sometimes it's important but does not imply they are similar items that should be linked in a kNN graph. It depends on context, which is beyond the scope of a generic kNN package.

This package can't redefine "correlation distance" to mean something different from how it is defined in the literature and other packages. It is simple to write your own metric if you want your distances to work that way, but it won't be correlation distance.

from pynndescent.

Well in that case, it might be worth including one sentence on this issue: Defining correlation "distance" as 1-corr does not treat negative correlation as nearest Neighbour.

from pynndescent.