This repository contains an ELMo and a word2vec model pre-trained on a 1 Billion word chemical patent corpus.

The dataset we used for training consists of 84,076 patent documents from 7 different patent authorities. The dataset will NOT be made publicly available.

We trained the word2vec model with same hyper-parameters in Pyysalo et al., (2013) for 10 iterations. The word vectors file (.txt) can be directly loaded into your neural network framework. Note that words shorter than 25 characters in length were replaced by long_token during training.

Please click here to download the pre-trained word vectors.

Default hyper-parameters in Peters et al., (2018) were used. Note that words shorter than 25 characters in length were replaced by long_token during training (cf. max. character length is 50 under default setting).

Please click here to download ELMo model.

• Fine Tuning: Load weights.hdf5 and options.json into the original ELMo implementation and train it further on your own datasets.

• Representation: You can also use contextualized word representations generated by ELMo for downstream tasks by load weights.hdf5 and options.json into AllenNLP framework.

If you find the word representations useful, please cite the following paper: Improving Chemical Named Entity Recognition in Patents with Contextualized Word Embeddings

@inproceedings{zhai2019improving,
  author    = {Zhai, Zenan and Nguyen, Dat Quoc and A. Akhondi, Saber and Thorne, Camilo and Druckenbrodt,
               Christian and Cohn, Trevor and Gregory, Michelle and Verspoor, Karin},
  title     = {{Improving Chemical Named Entity Recognition in Patents with Contextualized Word Embeddings}},
  booktitle = {{Proceedings of the BioNLP 2019 workshop}},
  pages     = {To appear},
  year      = {2019},
}