• Input: Bangla Text with mask. i.e.: আমার সোনার বাংলা < blank > তোমায় ভালবাসি
• Output: Input Text with Predicted mask. i.e.: আমার সোনার বাংলা আমি তোমায় ভালবাসি

This repository contains the code and resources for a BERT (Bidirectional Encoder Representations from Transformers) model trained on a Bangla language dataset for Masked Language Modeling ( we can use any bangla .txt format dataset). Masked Language Model (MLM) involves providing BERT with a sentence and fine-tuning its weights to produce the same sentence as output. Prior to presenting the input sentence to BERT, masking done in certain tokens. Thus, the BERT model is used to generate the original sentence after receiving the masked input.

Figure: In this image, before passing tokens into BERT — the lincoln token have masked, replacing it with [MASK] (Here in this project I use < blank >).

├── images
├── dataset
├── bert_module.py
├── config.py
├── data_preprocess.py
├── inference.py
├── logs
│   ├── bert_mlm_bangla.keras
│   ├── fit
│   └── vectorizer_layer.pkl
├── README.md
├── environment.yml
├── train.py
└── utils.py

To use the Bangla BERT-MLM model in your project, follow these steps:

git clone https://github.com/kamrul-brur/BERT-Bangla-MLM.git

• Create virtual environment and install dependiencies with the below command

conda env create -f environment.yml

• Then activate the environment

conda activate bert_mlm

In the config.py file, update the dataset path, log directory, model name, vectorizer layer name, max_epoch, and any other necessary parameters for training. Default values have been provided.

    MAX_LEN = 256
    BATCH_SIZE = 32
    LR = 0.001
    VOCAB_SIZE = 10000
    EMBED_DIM = 128
    NUM_HEAD = 8
    FF_DIM = 128
    NUM_LAYERS = 1
    EPOCHS = 100
    DATASET_PATH = "./dataset/dataset.txt"
    LOG_DIRECTORY = "logs"
    SAVED_MODEL_NAME = "bert_mlm_bangla.keras"
    SAVED_VECTORIZED_LAYER_NAME = "vectorizer_layer.pkl"
    TENSORBOARD_LOG_DIR = "logs/fit"

• Run the train.py file using the command

python train.py

The training will be completed and log file is saved on the directory /logs.

• Traning metrics in tensorboard can be visualized by the following command.

tensorboard -logdir logs

Then click on the url given in terminal to monitor performance in tensorboard

• Write your input masked sentences in a txt file. i.e.: inputs.txt
• In the ineference.py file, change the path of the input file to accordingly.

input_text_path = "dataset/inputs.txt"

• change the path of model and vectorizer_layer files if needed.

vectorized_layer_path = os.path.join(config.LOG_DIRECTORY, config.SAVED_VECTORIZED_LAYER_NAME)
model_path = os.path.join(config.LOG_DIRECTORY, config.SAVED_MODEL_NAME)

• Run the train.py file using the command

python ineference.py file

• Keras-team
• Bangla-BERT
• Masked-Language Modeling With BERT
• Huggingface-BERT

@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}