KV cache

In short: The KV cache stores the intermediate results (keys and values) of the tokens so far, so the model does not have to recompute them every time.

In more detail: The cache grows with every token. With long contexts it can take several gigabytes and must fit in memory in addition to the model. Modern models with “grouped-query attention” shrink it considerably.

In Depth

• Size: 2 × layers × KV heads × head size × bytes per value × tokens. An 8-billion model with 32 layers needs about 0.13 MB per token in FP16, about 4 GB at 32,000 tokens.
• Quantised: The cache can be stored in 8-bit, halving the need.
• Effect on speed: With long contexts each new token also reads the whole cache. That slows things down.
• Reuse: A shared beginning of several prompts (system prompt) can stay in the cache (“prompt caching”).

See also: Context window, VRAM and model size