Context window

In short: The context window is the amount of text (in tokens) a language model can consider at once: question, history so far, documents and answer together.

In more detail: Older models had 2,000 to 8,000 tokens, current ones 128,000 to over a million. The larger the window, the more memory the KV cache needs and the slower each new token becomes.

In Depth

• Lost in the middle: Models often use information at the beginning and end of long texts better than in the middle.
• No memory: What is not in the window, the model does not know. Applications therefore send the history along or summarise it.
• RAG: Instead of putting everything into the window, RAG fetches only the matching text pieces.
• Cost: Long prompts cost time and, with cloud providers, money.

See also: Token (AI), RAG (retrieval-augmented generation)