Prefill and decode

In short: A language model works in two phases: prefill processes the whole prompt at once, decode generates the answer token by token.

In more detail: Prefill is compute-heavy and parallelises well, so it is much faster per token (hundreds to thousands of tokens per second). Decode is limited by memory bandwidth and runs only as fast as a single read cycle through the model.

Our context: The AI speed calculator shows both values: how fast the prompt is processed and how fast the answer appears.

In Depth

• Time to first token: The wait until the first word equals the prefill time. Long documents extend it.
• Compute vs. bandwidth: Prefill needs TFLOPS, decode needs GB/s.
• Batching: Several requests at once use bandwidth better because the weights are read only once.
• Context: The longer the context, the more KV cache has to be read per token.

See also: Tokens per second, KV cache