Table of Contents generated with DocToc

• Awesome-Speech-Pretraining
  • Papers
    • 2018
    • 2019
    • 2020
    • 2021
    • 2022
    • Speech + Text
    • SSL for TTS
  • Resources
  • Statistics
    • wav2vec 2.0
      • Pre-training
      • Fine-tuning
    • wav2vec-u
    • HuBERT
      • Pre-training
      • Fine-tuning
    • data2vec

Papers, Resources, and Statistics for Self-Supervised Learning and Pre-Training on Speech.

🌟 represents important papers.

• 🌟 CPC: Representation Learning with Contrastive Predictive Coding - A Oord et al, arXiv 2018

• APC: An Unsupervised Autoregressive Model for Speech Representation Learning - YA Chung et al, INTERSPEECH 2019
• 🌟 wav2vec: Unsupervised Pre-training for Speech Recognition - S Schneider et al, INTERSPEECH 2019
• 🌟 vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations - A Baevski et al, arXiv 2019, ICLR 2020
• MPC: Improving Transformer-based Speech Recognition Using Unsupervised Pre-training - D Jiang et al, arXiv 2019
• PASE: Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks - S Pascual et al, INTERSPEECH 2019

• Bidir CPC: Learning robust and multilingual speech representations - K Kawakami et al, EMNLP 2020
• Multi-target APC: Improved speech representations with multi-target autoregressive predictive coding - YA Chung et al, ACL 2020
• Modified CPC: Unsupervised pretraining transfers well across languages - M Riviere et al, ICASSP 2020
• Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders - AT Liu et al, ICASSP 2020
• vq-wav2vec-FT: Effectiveness of self-supervised pre-training for asr - A Baevski et al, ICASSP 2020
• DeCoAR: Deep contextualized acoustic representations for semi-supervised speech recognition - S Ling et al, ICASSP 2020
• Improved noisy student training for automatic speech recognition - DS Park et al, INTERSPEECH 2020
• 🌟 wav2vec 2.0: A framework for self-supervised learning of speech representations - A Baevski et al, NeurIPS 2020
• Multi-lingual wav2vec 2.0: Unsupervised cross-lingual representation learning for speech recognition - A Conneau et al, arXiv 2020
• Self-Training wav2vec 2.0: Self-training and Pre-training are Complementary for Speech Recognition - Q Xu et al, arXiv 2020, ICASSP 2021
• Decoar 2.0: Deep contextualized acoustic representations with vector quantizationarXiv 2020
• Pushing the limits of semi-supervised learning for automatic speech recognition - Y Zhang et al, arXiv 2020

• Unispeech: Unified speech representation learning with labeled and unlabeled data- C Wang et al, ACL 2021

• Tera: Self-supervised learning of transformer encoder representation for speech - AT Liu et al, TASLP 2020

• Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training - WN Hsu et al, INTERSPEECH 2021

• Zero-shot wav2vec 2.0: Simple and Effective Zero-shot Cross-lingual Phoneme Recognition - Q Xu et al, arXiv 2021

• 🌟 wav2vec-U: Unsupervised Speech Recognition - A Baevski et al, NeurIPS 2021

• 🌟 HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units - WN Hsu et al, TASLP 2021

• 🌟 SUPERB: Speech processing Universal PERformance Benchmark - S Yang et al, INTERSPEECH 2021

• Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing - J Ao et al, arXiv 2021

• Wavlm: Large-scale self-supervised pre-training for full stack speech processing - S Chen et al, arXiv 2021

• Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition - Y Zhang et al, arXiv 2021

• Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition - G Zheng et al, EMNLP 2021

• Data2vec: A general framework for self-supervised learning in speech, vision and language - A Baevski et al, arXiv 2022
• BEST-RQ: Self-supervised Learning with Random-projection Quantizer for Speech Recognition - CC Chiu et al, arXiv 2022
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities - HS Tsai et al, arXiv 2022
• wav2vec-U 2.0: Towards End-to-end Unsupervised Speech Recognition - AH Liu et al, arXiv 2022
• c-siam: Contrastive Siamese Network for Semi-Supervised Speech Recognition - S Khorram et al, ICASSP 2022
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages - F Wu et al, arXiv 2022
• Speech2C: Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data - J Ao et al, arXiv 2022

• A general multi-task learning framework to leverage text data for speech to text tasks - Y Tang et al, ICASSP 2021
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training - A Bapna et al, arXiv 2021
• mSLAM: Massively multilingual joint pre-training for speech and text - A Bapna et al, arXiv 2022
• Optimizing Alignment of Speech and Language Latent Spaces for End-to-End Speech Recognition and Understanding - W Wang et al, INTERSPEECH 2022
• STPT: Unified Speech-Text Pre-training for Speech Translation and Recognition - Y Tang et al, ACL 2022
• Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data - Y Kang et al, AAAI 2022

• Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks - R Eloff et al, INTERSPEECH 2019
• Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages - H Zhang et al, INTERSPEECH 2020

Speech processing Universal PERformance Benchmark (SUPERB)

Self-Supervised Speech Pre-training and Representation Learning (S3PRL)

Statistics on speech pretraining.