Long contexts: techniques and limits

In short: Modern models process hundreds of thousands to millions of tokens. This works with adapted positional encodings, efficient attention and lots of memory. Even so, models do not use long contexts equally well throughout.

In more detail: Attention grows quadratically with length and the KV cache grows linearly. Both must stay manageable.

In detail

• Training: first short, then step by step further trained with longer examples, with stretched RoPE frequencies, see Positional encoding and RoPE.
• Efficiency: FlashAttention, sliding window, GQA/MLA (Multi-head attention and variants), quantisation of the KV cache.
• Ring attention: distributes long sequences across several GPUs.
• Needle in a haystack: test whether a single sentence is found in the long text. Good models pass, but real tasks (connections across many places) are harder.
• “Lost in the middle”: information in the middle is used worse than at the start and end.
• Cost: every token in the context costs compute and memory at every answer. Caching (Inference servers: batching, PagedAttention and vLLM) and targeted selection via RAG save.
• Practice: put only relevant things into the context, important things at the start or end, summarise long conversations.

See also: Context window, KV cache