FastSeq provides efficient implementation of popular sequence models (e.g. Bart, ProphetNet) for text generation, summarization, translation tasks etc. It automatically optimizes inference speed based on pupular NLP toolkits (e.g. FairSeq and HuggingFace-Transformers) without accuracy loss. All these can be easily done (no need to change any code/model/data if using our command line tool, or simply add one-line code import fastseq
if using source code).
Below shows the generation speed gain by using FastSeq.
Model | W/O FastSeq (in samples/s) | W/ FastSeq (in samples/s) | Speedup |
---|---|---|---|
ProphetNet | 2.8 | 11.3 | 4.0x |
Bart (fs ) |
2.7 | 15.0 | 5.6x |
Bart (hf ) |
3.5 | 11.4 | 3.3x |
DistilBart (hf ) |
4.3 | 13.8 | 3.2x |
T5 (hf ) |
5.0 | 11.5 | 2.3x |
WMT16 En-De (fs ) |
84.0 | 135.0 | 1.6x |
- All benchmarking experiments run on NVIDIA-V100-16GB with docker. Highest speed recorded for each model by tuning batch size. For parameter setting details, click link of corresponding model.
fs
stands for Fairseq 0.9.0 version,hf
stands for Huggingface Transformers 3.0.2 version.- Optimizations were automatically applied to all generation/sequence models in Fairseq & Huggingface Transformers. Above only lists a subset of them.
We developped a wide range of speedup techniques, including improving beam search efficiency, reducing memory footprint, speeding up calculation for key operations etc, IO speedup etc. To seamlessly connect with community, they were applied to existing models from Fairseq and Huggingface Transformers in the backend, while keeping model interface and usage same as before.
- Python version >= 3.6
- torch >= 1.4.0
- fairseq == 0.9.0
- transformers == 3.0.2
- requets >= 2.24.0
- absl-py >= 0.9.0
- rouge-score >= 0.0.4
If you use fairseq or transformers, you only need to install one of them. If you use both, you need to install both.
# when fairseq and/or transformers has been installed
$ pip install git+https://github.com/microsoft/fastseq.git
# install fastseq + transformers
$ pip install git+https://github.com/microsoft/fastseq.git#egg=project[transformers]
# install fastseq + fairseq
$ pip install git+https://github.com/microsoft/fastseq.git#egg=project[fairseq]
# install fastseq + transformers + fairseq
$ pip install git+https://github.com/microsoft/fastseq.git#egg=project[transformers,fairseq]
Only one line of code change is needed to use the optimizations provided by FastSeq
.
# import fastseq at the beginning of your program
import fastseq
import torch
# Download bart.large.cnn
bart = torch.hub.load('pytorch/fairseq', 'bart.large.cnn')
bart.cuda() # use GPU
bart.eval() # disable dropout for evaluation
bart.half()
slines = ['FastSeq provides efficient implementations of the popular sequence models. Please visit https://github.com/microsoft/fastseq for more details.']
hypotheses = bart.sample(
slines, beam=4, lenpen=2.0, max_len_b=140, min_len=55, no_repeat_ngram_size=3)
print(hypotheses)
Example usage for bart model on cnn daily mail task.
$ fastseq-generate-for-fairseq \
cnn_dnn/bin \
--path bart.large.cnn/model.pt \
--fp16 \
--task translation \
--batch-size 128 \
--gen-subset valid \
--truncate-source \
--bpe gpt2 \
--beam 4 \
--num-workers 4 \
--min-len 55 \
--max-len-b 140 \
--no-repeat-ngram-size 3 \
--lenpen 2.0
Both model file and task data file are the same as original Fairseq version.
Example usage for bart model on cnn daily mail task.
$ fastseq-generate-for-transformers \
facebook/bart-large-cnn \
cnn_dm/val.source \
out.summary \
--reference_path cnn_dm/val.target \
--device cuda \
--bs 128 \
--fp16 \
--score_path out.score \
--task summarization
Both model file and task data file are the same as original Transformers version.
# run a single test.
$ python tests/optimizer/fairseq/test_fairseq_optimizer.py
# run all the tests.
$ python -m unittest discover -s tests/ -p '*.py'
# run all the benchmarks.
$ cd benchmarks && bash run_all_benchmarks.sh
Changes to Python code should conform to PEP 8. yapf
can be used to help format the python code, and use pylint
to check your Python changes.
# format the code by yapf
$ yapf --style pep8 -i -r PYTHON_FILE/PACKAGE
# run pylint check
$ pylint --rcfile=.pylintrc PYTHON_FILE/PACKAGE
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.