Use spaCy to go beyond vanilla word2vec

Read about sense2vec here:

https://spacy.io/blog/sense2vec-with-spacy

You can use an online demo of the technology here:

https://sense2vec.spacy.io/

We're currently refining the API, to make this technology easy to use. Once we've completed that, you'll be able to download the package on PyPi. For now, the code is available to clarify the blog post.

There are three relevant files in this repository:

This script pre-processes text using spaCy, so that the sense2vec model can be trained using Gensim.

This script reads a directory of text files, and then trains a word2vec model using Gensim. The script includes its own vocabulary counting code, because Gensim's vocabulary count is a bit slow for our large, sparse vocabulary.

To serve the similarity queries, we wrote a small vector-store class in Cython. This made it easier to add an efficient cache in front of the service. It also less memory than Gensim's Word2Vec class, as it doesn't hold the keys as Python unicode strings.

Similarity queries could be faster, if we had made all vectors contiguous in memory, instead of holding them as an array of pointers. However, we wanted to allow a .borrow() method, so that vectors can be added to the store by reference, without copying the data.