fredriko / bert-tensorflow-pytorch-spacy-conversion Goto Github PK

This repository contains instructions and sample code for converting a BERT Tensorflow model to work with Hugging Face's pytorch-transformers and as a package for explosion.ai's spaCy via spacy-pytorch-transformers.

The instructions use the Russian BERT model (RuBERT) created by DeepPavlov as working example.

You need the following software installed on your computer to be able to install and run the examples in this guide.

• Git
• Python 3.6 or later
• pip3
• virtualenv

Download RuBERT from http://files.deeppavlov.ai/deeppavlov_data/bert/rubert_cased_L-12_H-768_A-12_v1.tar.gz

For the sake of this example, place the downloaded RuBERT file in your user's root directory, and unpack it with

tar zxvf rubert_cased_L-12_H-768_A-12_v1.tar.gz

The unpacked model is now available in ~/rubert_cased_L-12_H-768_a-12_v1/

Clone this repository, create a virtual environment, and install the dependencies by giving the following commands in a shell:

git clone https://github.com/fredriko/bert-tensorflow-pytorch-spacy-conversion.git
cd bert-tensorflow-pytorch-spacy-conversion
virtualenv -p python3 ~/venv/bert-tensorflow-pytorch-spacy-conversion
source ~/venv/bert-tensorflow-pytorch-spacy-conversion/bin/activate
pip3 install -r requirements.txt

Convert the Tensorflow RuBERT model to a PyTorch equivalent with this command:

$ python3 -m pytorch_transformers.convert_tf_checkpoint_to_pytorch \
--tf_checkpoint_path ~/rubert_cased_L-12_H-768_A-12_v1/bert_model.ckpt.index \ 
--bert_config_file ~/rubert_cased_L-12_H-768_A-12_v1/bert_config.json \
--pytorch_dump_path ~/rubert_cased_L-12_H-768_A-12_v1/pytorch_model.bin

After the conversion, copy the required files to a separate directory; ~/pytorch-rubert/:

mkdir ~/pytorch-rubert
cp ~/rubert_cased_L-12_H-768_A-12_v1/rubert_pytorch.bin ~/pytorch-rubert/.
cp ~/rubert_cased_L-12_H-768_A-12_v1/vocab.txt ~/pytorch-rubert/.
cp ~/rubert_cased_L-12_H-768_A-12_v1/bert_config.json ~/pytorch-rubert/config.json

You now have the files required to use RuBERT in pytorch-transformers. The following code snippet is an example of how the PyTorch model can be loaded and used in pytorch-transformers (source):

import torch
from pytorch_transformers import *
from pathlib import Path

sample_text = "Рад познакомиться с вами."
my_model_dir = str(Path.home() / "pytorch-rubert")

tokenizer = BertTokenizer.from_pretrained(my_model_dir)
model = BertModel.from_pretrained(my_model_dir, output_hidden_states=True)

input_ids = torch.tensor([tokenizer.encode(sample_text, add_special_tokens=True)])
print(f"Input ids: {input_ids}")
with torch.no_grad():
    last_hidden_states = model(input_ids)[0]
    print(f"Shape of last hidden states: {last_hidden_states.shape}")
    print(last_hidden_states)

In order to create a spaCy package of the PyTorch model, it first has to be saved to disk as a serialized pipeline. First, create the directory in which to save the pipeline, then run the script for serializing and saving it.

mkdir ~/spacy-rubert
python3 -m src.serialize_spacy_nlp_pipeline

You now have all you need to create a spaCy package in ~/spacy-rubert.

OPTIONAL: fill in the appropriate information in ~/spacy-rubert/meta.json before proceeding.

Run the following commands to create a spaCy package from the serialized pipeline and save it to ~/spacy-rubert-package:

mkdir ~/spacy-rubert-package
python3 -m spacy package ~/spacy-rubert ~/spacy-rubert-package

NOTE: that the name of the model directory under ~/spacy-rubert-package depends on the information you supplied in ~/spacy-rubert/meta.json in the previous step. The name used below originates from a raw meta.json file.

cd ~/spacy-rubert-package/ru_model-0.0.0
python3 setup.py sdist

After successful completion of the above commands, the RuBERT model is available as a spaCy package in:

~/spacy-rubert-package/ru_model-0.0.0/dist/ru_model-0.0.0.tar.gz

Install it with:

pip3 install ~/spacy-rubert-package/ru_model-0.0.0/dist/ru_model-0.0.0.tar.gz

Verify its presence in the current virtualenv:

pip3 freeze | grep ru-model
> ru-model==0.0.0

Here is an example of how the package can be loaded and used (source):

import spacy

nlp = spacy.load("ru_model")
doc = nlp("Рад познакомиться с вами.")
print(doc.vector)
print(doc[0].similarity(doc[0]))
print(doc[0].similarity(doc[1]))

NOTE: that the above example does not make use of a GPU. For that to happen, you need a different installation of spaCy than the one specified in the requirements.txt in this repository.