Efficient Computation of Similarity Matrix for Set of Documents? about scispacy HOT 3 CLOSED

Hi!

To do this faster, you should batch them together and compute the result using a matrix multiplication, which you can speed up by using a GPU, for example.

More generally you should be a little careful of relying on the document vectors of spacy models, as I think they are just word vector averages. Also note that the scispacy med model only has 20k vectors, which is not a huge amount, so many words will be OOV (i.e not have a vector). Is a large model (~2GB with vectors) something that you would be interested in?

from scispacy.

Thanks for reply. This post on stackexchange which uses matrix multiplication via numpy is significantly boosting speed: https://stackoverflow.com/questions/41905029/create-cosine-similarity-matrix-numpy?rq=1

I would be interested in larger vector models. I realize there are caveats to using them but am trying to pair them with other features to cluster/classify biomedical documents.

from scispacy.

Cool, i'll bear that in mind for what to work on for the next release!

from scispacy.