Awesome Pretrained Chinese NLP Models

在自然语言处理领域中，预训练语言模型（Pretrained Language Models）已成为非常重要的基础技术，本仓库主要收集了目前网上公开的一些高质量中文预训练模型(感谢分享资源的大佬)，并将持续更新......

注: 🤗huggingface模型下载地址: 1. 清华大学开源镜像 2. 官方地址

Expand Table of Contents

• NLU系列
  • BERT
  • RoBERTa
  • ALBERT
  • NEZHA
  • XLNET
  • MacBERT
  • ELECTRA
  • ZEN
  • ERNIE
• NLG系列
  • GPT
  • NEZHA-GEN

NLU系列

BERT

• 2018 | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Jacob Devlin, et al. | arXiv | PDF
• 2019 | Pre-Training with Whole Word Masking for Chinese BERT | Yiming Cui, et al. | arXiv | PDF

备注:

[1] wwm全称为**Whole Word Masking **,一个完整的词的部分WordPiece子词被mask，则同属该词的其他部分也会被mask

[2] ext表示在更多数据集下训练

RoBERTa

• 2019 | RoBERTa: A Robustly Optimized BERT Pretraining Approach | Yinhan Liu, et al. | arXiv | PDF

ALBERT

• 2019 | ALBERT: A Lite BERT For Self-Supervised Learning Of Language Representations | Zhenzhong Lan, et al. | arXiv | PDF

NEZHA

• 2019 | NEZHA: Neural Contextualized Representation for Chinese Language Understanding | Junqiu Wei, et al. | arXiv | PDF

MacBERT

• 2020 | Revisiting Pre-Trained Models for Chinese Natural Language Processing | Yiming Cui, et al. | arXiv | PDF

XLNET

• 2019 | XLNet: Generalized Autoregressive Pretraining for Language Understanding | Zhilin Yang, et al. | arXiv | PDF

ELECTRA

• 2020 | ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators | Kevin Clark, et al. | arXiv | PDF

ZEN

• 2019 | ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations | Zhilin Yang, et al. | arXiv | PDF

ERNIE

• 2019 | ERNIE: Enhanced Representation through Knowledge Integration | Yu Sun, et al. | arXiv | PDF

• 2020 | SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis | Hao Tian, et al. | arXiv | PDF

备注:

[1] PaddlePaddle转TensorFlow可参考: tensorflow_ernie

[2] PaddlePaddle转PyTorch可参考: ERNIE-Pytorch

NLG系列

GPT

• 2019 | Improving Language Understandingby Generative Pre-Training | Alec Radford, et al. | arXiv | PDF

• 2019 | Language Models are Unsupervised Multitask Learners | Alec Radford, et al. | arXiv | PDF

NEZHA-Gen