Traditional language models like ChatGPT have a fixed context window, which means they can only consider a limited number of tokens (words or subwords) as input to generate the next word in a sequence. For example, the original GPT-3.5 has a maximum context window of 2,048 tokens. This limitation poses challenges when dealing with longer pieces of text, as it may cut off relevant information beyond the context window.

To overcome this limitation and better model long-range dependencies in text, researchers have explored various techniques and architectures. The following are some approaches that might be considered as "long context language models".

Some language models incorporate external memory mechanisms, allowing them to store information from past tokens and retrieve it when necessary. These memories enable the model to maintain context over longer segments of text.

• Dual Cache for Long Document Neural Coreference Resolution
• Augmenting Language Models with Long-Term Memory
• In-context Autoencoder for Context Compression in a Large Language Model
• LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models

Hierarchical Models / Data-Centric or Compress happens in context / key-value

• Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading
• Compressing Context to Enhance Inference Efficiency of Large Language Models
• Context Compression for Auto-regressive Transformers with Sentinel Tokens
• Learned Token Pruning for Transformers
• Block-Recurrent Transformers
• Recurrent Memory Transformer

Probably transformer variants, which means that the architecture of the transformers

• Memorizing Transformers
• Compressive Transformers for Long-Range Sequence Modelling
• Focused Transformer: Contrastive Training for Context Scaling

Researchers have explored variants that can better handle long context by utilizing techniques like sparse attention, axial attention, and reformulating the self-attention mechanism.

• Adapting Language Models to Compress Contexts
• LONGNET: Scaling Transformers to 1,000,000,000 Tokens
• Blockwise Parallel Transformer for Long Context Large Models
• ETC: Encoding Long and Structured Inputs in Transformers
• Improving Long Context Document-Level Machine Translation
• Extending context window of large language models via positional interpolation

Rather than relying on a fixed context window, some models use a sliding window approach. They process the text in smaller chunks, capturing local dependencies within each window and passing relevant information between adjacent windows.

• Efficient Long-Text Understanding with Short-Text Models
• LongCoder: A Long-Range Pre-trained Language Model for Code Completion
• Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
• Train short, test long: Attention with linear biases enables input length extrapolation

• Lost in the Middle: How Language Models Use Long Contexts
• Do Long-Range Language Models Actually Use Long-Range Context?
• Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?

Some approaches use reinforcement learning to guide the model's attention to focus on important parts of the input text while considering the context.

• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

• L-Eval: Instituting Standardized Evaluation for Long Context Language Models
• Long Range Arena: A Benchmark for Efficient Transformers
• Large-Scale Language Model Rescoring on Long-Form Data

• XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
• Visual Transformers needs Registers

If you have any questions or comments, please feel free to let us know: 📧 Cheng Deng. The main contribution of the contributor:

• Cheng Deng @ SJTU