“李太白少时，梦所用之笔头上生花后天才赡逸，名闻天下。”——王仁裕《开元天宝遗事·梦笔头生花》

Docs | Model | Dataset | Paper | 中文版

TextBox is developed based on Python and PyTorch for reproducing and developing text generation algorithms in a unified, comprehensive and efficient framework for research purpose. Our library includes 21 text generation algorithms, covering two major tasks:

• Unconditional (input-free) Generation
• Conditional (Seq2Seq) Generation, including Machine Translation, Text Summarization, Attribute-to-Text, and Dialogue Systems

We provide the support for 9 benchmark text generation datasets. A user can apply our library to process the original data copy, or simply download the processed datasets by our team.

Figure: The Overall Architecture of TextBox

• Unified and modularized framework. TextBox is built upon PyTorch and designed to be highly modularized, by decoupling diverse models into a set of highly reusable modules.
• Comprehensive models, benchmark datasets and standardized evaluations. TextBox also contains a wide range of text generation models, covering the categories of VAE, GAN, RNN or Transformer based models, and pre-trained language models (PLM).
• Extensible and flexible framework. TextBox provides convenient interfaces of various common functions or modules in text generation models, RNN encoder-decoder, Transformer encoder-decoder and pre-trained language model.
• Easy and convenient to get started. TextBox provides flexible configuration files, which allows green hands to run experiments without modifying source code, and allows researchers to conduct qualitative analysis by modifying few configurations.

TextBox requires:

• Python >= 3.6.2

• torch >= 1.6.0. Please follow the official instructions to install the appropriate version according to your CUDA version and NVIDIA driver version.

• GCC >= 5.1.0

pip install textbox

If you face a problem when installing fast_bleu, for Linux, please ensure GCC >= 5.1.0. For Windows, you can use the wheels in fast_bleu_wheel4windows for installation. For MacOS, you can install with the following command:

pip install fast-bleu --install-option="--CC=<path-to-gcc>" --install-option="--CXX=<path-to-g++>"

After installing fast_bleu successfully, just reinstall textbox.

git clone https://github.com/RUCAIBox/TextBox.git && cd TextBox
pip install -e . --verbose

With the source code, you can use the provided script for initial usage of our library:

python run_textbox.py

This script will run the RNN model on the COCO dataset to conduct unconditional generation. Typically, this example takes a few minutes. We will obtain the output log like example.log.

If you want to change the parameters, such as rnn_type, max_vocab_size, just set the additional command parameters as you need:

python run_textbox.py --rnn_type=lstm --max_vocab_size=4000

We also support to modify YAML configuration files in corresponding dataset and model properties folders and include it in the command line.

If you want to change the model, the dataset or the task type, just run the script by modifying corresponding command parameters:

python run_textbox.py --model=[model_name] --dataset=[dataset_name]

model_name is the model to be run, such as RNN and BART. Models we implemented can be found in Model.

If you want to change the datasets, please refer to Dataset.

If TextBox is installed from pip, you can create a new python file, download the dataset, and write and run the following code:

from textbox.quick_start import run_textbox

run_textbox(config_dict={'model': 'RNN',
                         'dataset': 'COCO',
                         'data_path': './dataset'})

This will perform the training and test of the RNN model on the COCO dataset.

If you want to run different models, parameters or datasets, the operations are same with Start from source.

TextBox supports to apply part of pretrained language models (PLM) to conduct text generation. Take the GPT-2 for example, we will show you how to use PLMs to fine-tune.

1. Download the GPT-2 model provided from Hugging Face (https://huggingface.co/gpt2/tree/main), including config.json, merges.txt, pytorch_model.bin, tokenizer.jsonand vocab.json. Then put them in a folder at the same level as textbox, such as pretrained_model/gpt2.

2. After downloading, you just need to run the command:

python run_textbox.py --model=GPT2 --dataset=COCO \
                      --pretrained_model_path=pretrained_model/gpt2

TextBox supports to train models with multiple GPUs conveniently. You don't need to modify the model, just run the following command:

python -m torch.distributed.launch --nproc_per_node=[gpu_num] \
       run_textbox.py --model=[model_name] \
       --dataset=[dataset_name] --gpu_id=[gpu_ids] --DDP=True

gpu_num is the number of GPUs you want to train with (such as 4), and gpu_ids is the usable GPU id list (such as 0,1,2,3).

Notice that: we only support DDP for end-to-end model. We will add support for non-end-to-end models, such as GAN, in the future.

The above Figure presents the overall architecture of our library. The running procedure relies on some experimental configuration, obtained from the files, command line or parameter dictionaries. The dataset and model are prepared and initialized according to the configured settings, and the execution module is responsible for training and evaluating models. The details of interfaces can be obtained in our document.

We implement 21 text generation models, covering unconditional generation and sequence-to-sequence generation, in the following table:

We have also collected 9 datasets that are commonly used for six tasks, which can be downloaded from Google Drive and Baidu Wangpan (Password: lwy6), including raw data and processed data.

We list the 9 datasets in the following table:

We also support you to run our model using your own dataset. Just follow the three steps:

1. Create a new folder under the dataset folder to put your own corpus file which includes a sequence per line, e.g. dataset/YOUR_DATASET;

2. Write a YAML configuration file using the same file name to set the hyper-parameters of your dataset, e.g. textbox/properties/dataset/YOUR_DATASET.yaml.

   If you want to splitted the dataset, please set split_strategy: "load_split" in the yaml, just as the COCO yaml or IWSLT14_DE_EN yaml.

   If you want to split the dataset by ratio automaticly, please set split_strategy: "by_ratio" and your desired split_ratio in the yaml, just as the IMDB yaml.

3. For unconditional generation, name the corpus file corpus.txt if you set"by_ratio", name the corpus files train.txt, valid.txt, dev.txt if you set "load_split".

   For sequence-to-sequence generation, please name the corpus files train.[xx/yy], valid.[xx/yy], dev.[xx/yy], and the xx or yy is the suffix of the source or target file which should be consistent with source_suffix and target_suffix in the YAML.

We have implemented various text generation models, and compared their performance on unconditional and conditional text generation tasks. We also show a few generated examples, and more examples can be found in generated_examples.

The following results were obtained from our TextBox in preliminary experiments. However, these algorithms were implemented and tuned based on our understanding and experiences, which may not achieve their optimal performance. If you could yield a better result for some specific algorithm, please kindly let us know. We will update this table after the results are verified.

Negative Log-Likelihood (NLL), BLEU and Self-BLEU (SBLEU) on test dataset:

Part of generated examples:

NLL, BLEU and SBLEU on test dataset:

Part of generated examples:

NLL, BLEU and SBLEU on test dataset:

Part of generated examples (with max_length=100):

ROUGE metric on test dataset using beam search (with beam_size=5):

BLEU metric on test dataset with three decoding strategies: top-k sampling, greedy search and beam search (with beam_size=5):

BLEU and distinct metrics on test dataset using beam search (with beam_size=5):

BLEU and distinct metrics on test dataset using beam search (with beam_size=5):

Please let us know if you encounter a bug or have any suggestions by filing an issue.

We welcome all contributions from bug fixes to new features and extensions.

We expect all contributions discussed in the issue tracker and going through PRs.

We thank @LucasTsui0725 for contributing HRED model and @Richar-Du for CVAE model.

If you find TextBox useful for your research or development, please cite the following paper:

@article{textbox,
    title={TextBox: A Unified, Modularized, and Extensible Framework for Text Generation},
    author={Junyi Li, Tianyi Tang, Gaole He, Jinhao Jiang, Xiaoxuan Hu, Puzhao Xie, Wayne Xin Zhao, Ji-Rong Wen},
    year={2021},
    journal={arXiv preprint arXiv:2101.02046}
}

TextBox is developed and maintained by AI Box.

TextBox uses MIT License.