Tokens per second

In short: Tokens per second (tok/s) measures how fast a language model generates text. The higher, the smoother the answer feels.

In more detail: To generate each single token, the hardware must read almost all active weights of the model from memory once. Speed is therefore usually limited by memory bandwidth, not by compute. Rule of thumb: tok/s ≈ bandwidth × efficiency ÷ model size in bytes.

Our context: The AI speed calculator calculates this for thousands of components and models, accounting for quantisation, mixture of experts, offloading to RAM and context length.

In Depth

• Example: An 8-billion model in 4-bit is about 4.5 GB. A graphics card with 1,000 GB/s and 60 % efficiency manages about 600 ÷ 4.5 ≈ 130 tok/s.
• Reading speed: A human reads about 4 to 8 tokens per second. Anything above is “faster than reading”.
• Prefill is different: Processing the prompt is compute-heavy and much faster per token (prefill and decode).
• Factors: quantisation, model size, context (KV cache), software, batch size.

See also: Memory bandwidth, Quantisation, Prefill and decode