In the domain of continual learning, there has been a significant paradigm shift from traditional methodologies to those that integrate foundation LMs. First, foundation LMs demonstrate enhanced generalization and transfer learning abilities across diverse tasks owing to their broad pre-training on large-scale datasets. The model has specialized transfer capability to quickly adapt to downstream tasks with only a few samples. Consequently, it is crucial to mitigate the degradation of both the zero-shot transfer and history task abilities in LMs while facilitating the acquisition of new skills. Second, due to the substantial number of parameters in foundation LMs, it is crucial to employ parameter-efficient techniques, such as prompt tuning and adapters, to update parameters without comprehensive retraining. Third, the foundation LMs possess the capability to follow instructions through instructional learning, enabling more dynamic and context-aware interactions.

This is an updating survey for Foundation Language Models-based Continual Learning, a constantly updated and extended version of the manuscript: "Recent Advances of Foundation Language Models-based Continual Learning: A Survey"

• Related Surveys
  • Continual Learning
  • Continual Learning for Computer Vision
  • Continual Learning for NLP
  • Continual Learning for Other Domains
• Offline Continual Learning
  • Domain-Incremental Learning
    • PLMs-based DIL
    • LLMs-based DIL
    • VLMs-based DIL
  • Task-Incremental Learning
    • PLMs-based TIL
    • LLMs-based TIL
    • VLMs-based TIL
  • Class-Incremental Learning
    • PLMs-based CIL
    • VLMs-based CIL
• Online Continual Learning
  • Hard Task Boundary
    • PLMs-based HTB
  • Blurry Task Boundary
    • PLMs-based BTB
    • VLMs-based BTB
• DATASET
• Reference

• Continual lifelong learning with neural networks: A review [paper]
• A Comprehensive Survey of Continual Learning: Theory, Method and Application [paper]
• Deep Class-Incremental Learning: A Survey [paper]
• Replay in Deep Learning: Current Approaches and Missing Biological Elements [paper]

• A Continual Learning Survey: Defying Forgetting in Classification Tasks [paper]
• Recent Advances of Continual Learning in Computer Vision: An Overview [paper]
• Online continual learning in image classification: An empirical survey [paper]
• Class-Incremental Learning: Survey and Performance Evaluation on Image Classification [paper]
• A comprehensive study of class incremental learning algorithms for visual tasks [paper]

• Continual Lifelong Learning in Natural Language Processing: A Survey [paper]
• Continual Learning of Natural Language Processing Tasks: A Survey [paper]

• A Survey on Incremental Update for Neural Recommender Systems [paper]
• Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks [paper]
• Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges [paper]

• Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation [paper]
• Learning to Solve NLP Tasks in an Incremental Number of Languages [paper]
• Toward Continual Learning for Conversational Agents [paper]
• DEMix Layers: Disentangling Domains for Modular Language Modeling [paper]
• Decouple knowledge from parameters for plug-and-play language modeling [paper]

• Continual Pre-training of Language Models [paper]

• Parameter-Efficient Transfer Learning for NLP [paper]
• Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning [paper]
• P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks [paper]
• Continual Learning in Task-Oriented Dialogue Systems [paper]
• Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks [paper]
• CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks [paper]
• Continual Prompt Tuning for Dialog State Tracking [paper]
• Continual Training of Language Models for Few-Shot Learning [paper]
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 [paper]

• ELLE: Efficient Lifelong Pre-training for Emerging Data [paper]

• CPPO: Continual Learning for Reinforcement Learning with Human Feedback [paper]
• COPR: Continual Learning Human Preference through Optimal Policy Regularization [paper]
• LAMOL: LAnguage MOdeling for Lifelong Language Learning [paper]
• RVAE-LAMOL: Residual Variational Autoencoder to Enhance Lifelong Language Learning [paper]
• Reformulating Domain Adaptation of Large Language Models as Adapt-Retrieve-Revise [paper]
• Continual Learning Under Language Shift [paper]
• Towards Continual Knowledge Learning of Language Models [paper]

• Efficient Continual Pre-training for Building Domain Specific Large Language Models [paper]
• EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data [paper]
• Adapting Large Language Models via Reading Comprehension [paper]
• Continual Pre-Training Mitigates Forgetting in Language and Vision [paper]

• Lifelong language pretraining with distribution-specialized experts [paper]

• Towards General Purpose Medical AI: Continual Learning Medical Foundation Model [paper]
• S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning [paper]
• VQACL: A Novel Visual Question Answering Continual Learning Setting [paper]

• Generative Replay Inspired by Hippocampal Memory Indexing for Continual Language Learning [paper]
• Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation [paper]
• Sentence Embedding Alignment for Lifelong Relation Extraction [paper]
• One Person, One Model, One World: Learning Continual User Representation without Forgetting [paper]
• Task Relation-aware Continual User Representation Learning [paper]
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning [paper]
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning [paper]
• MeLL: Large-scale Extensible User Intent Classification for Dialogue Systems with Meta Lifelong Learning [paper]
• Lifelong and Continual Learning Dialogue Systems: Learning during Conversation [paper]
• Lifelong and Continual Learning Dialogue System [paper]
• Learning on the Job: Online Lifelong and Continual Learning [paper]

• ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding [paper]
• Recyclable Tuning for Continual Pre-training [paper]

• Continual Sequence Generation with Adaptive Compositional Modules [paper]
• Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning [paper]

• Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue [paper]
• Progressive Prompts: Continual Learning for Language Models [paper]
• ConTinTin: Continual Learning from Task Instructions [paper]
• Large-scale Lifelong Learning of In-context Instructions and How to Tackle It [paper]

• From Static to Dynamic: A Continual Learning Framework for Large Language Models [paper]
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning [paper]
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [paper]
• Scalable Language Model with Generalized Continual Learning [paper]

• ConPET: Continual Parameter-Efficient Tuning for Large Language Models [paper]
• Exploring the Benefits of Training Expert Language Models over Instruction Tuning [paper]
• Orthogonal Subspace Learning for Language Model Continual Learning [paper]

• Fine-tuned Language Models are Continual Learners [paper]
• InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning [paper]

• CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation [paper]
• Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models [paper]

• Decouple Before Interact: Multi-Modal Prompt Learning for Continual Visual Question Answering [paper]
• CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model [paper]

• Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters [paper]

• Continual Learning for Named Entity Recognition [paper]
• Continual Learning for Sentence Representations Using Conceptors [paper]
• Continual Learning for Text Classification with Information Disentanglement Based Regularization [paper]

• Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection [paper]

• Continual Few-shot Intent Detection [paper]
• Rehearsal-free Continual Language Learning via Efficient Parameter Isolation [paper]
• Domain-Agnostic Neural Architecture for Class Incremental Continual Learning in Document Processing Platform [paper]

• VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model [paper]
• Learning without Forgetting for Vision-Language Models [paper]
• CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models [paper]
• Generative Multi-modal Models are Good Class-Incremental Learners [paper]

• Class Incremental Learning with Pre-trained Vision-Language Models [paper]

• Introducing Language Guidance in Prompt-based Continual Learning [paper]

• A Progressive Model to Enable Continual Learning for Semantic Slot Filling [paper]
• Online Continual Learning in Keyword Spotting for Low-Resource Devices via Pooling High-Order Temporal Statistics [paper]
• Rehearsal-Free Online Continual Learning for Automatic Speech Recognition [paper]

• Episodic Memory in Lifelong Language Learning [paper]
• Efficient Meta Lifelong-Learning with Limited Memory [paper]
• Meta-Learning with Sparse Experience Replay for Lifelong Language Learning [paper]
• Rehearsal-Free Online Continual Learning for Automatic Speech Recognition [paper]
• Lifelong Intent Detection via Multi-Strategy Rebalancing [paper]

• Episodic Memory in Lifelong Language Learning [paper]
• Efficient Meta Lifelong-Learning with Limited Memory [paper]
• Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking [paper]
• Meta-Learning Representations for Continual Learning [paper]
• Meta-Learning with Sparse Experience Replay for Lifelong Language Learning [paper]

• Continual Vision-Language Retrieval via Dynamic Knowledge Rectification [paper]

• CBA: Improving Online Continual Learning via Continual Bias Adaptor [paper]

• Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuning [paper]

If you find our survey or this collection of papers useful, please consider citing our work by

@article{yang2024recent,
  title={Recent Advances of Foundation Language Models-based Continual Learning: A Survey},
  author={Yang, Yutao and Zhou, Jie and Ding, Xuanwen and Huai, Tianyu and Liu, Shunyu and Chen, Qin and He, Liang and Xie, Yuan},
  journal={arXiv preprint arXiv:2405.18653},
  year={2024}
}