A study on encoding english sentences to tensorflow vectors or tensors using pre-trained BERT model from the Hugging Face Library.

BERT (Bi-directional Encoder representation from Transformers) is a modern language model which can be used for sentence-encoding purposes for a wide variety of applications like sentiment analysis, embeddings, chatbots, text classification, etc. developed by researchers at Google Language AI. BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks. Hence, BERT effectively paves a way for Transfer Learning in the domain of Natural Language Processing.

We have used the BERT implementations on the popular HuggingFace Library

• DistillBERT
• RoBERTa
• General Language Understanding Evaluation (GLUE)
• BERT-base
• BERT-large

The pre-training of a BERT model is done by 2 main methods :

• Masked Language Model (MLM)
• Next Sentence Prediction (NSP)

Then, the weights of the pre-trained model are fine-tuned as required for a particular use case. We can refer this paper to learn more about fine-tuning BERT for various specific tasks.

Here we have studied the uncased DistilBERT model in detail, and how to implement it in practice to obtain the standard 768-dimensional feature vectors of any English sentence encoded using the pre-trained model.

SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.

Here, we have examined in detail this BERT-based model fine-tuned for the specific task of Question Answering. We have used the SQuAD implementation on the Huggingface library.

• Mobile BERT QA System (code in pure TF/Keras)
• Testing BERT based Question Answering on Coronavirus articles
• Examples of fine-tuning BERT on custom-data
• HuggingFace NLP with Tensorflow 2.0
• HuggingFace IPython Notebooks
• Testing SQuAD with Amazon Rainforest Passage by Rajpurkar