Inference servers: batching, PagedAttention and vLLM

In short: An inference server answers many requests at once. With continuous batching, PagedAttention and caching it gets far more answers per second out of the same GPU.

In more detail: Because text generation is limited by memory bandwidth, it costs hardly any more time to process several requests at once: the model is read once and serves many users.

In detail

• Static batching: wait for the slowest request. Wastes performance.
• Continuous batching: finished requests are replaced immediately by new ones, the GPU stays busy.
• PagedAttention (vLLM): the KV cache is divided into small pages like virtual memory in an operating system. This avoids waste from fragmentation.
• Prefix caching: identical beginnings (system prompt, long documents) are computed only once. That is why cached input tokens are often much cheaper in APIs.
• Chunked prefill, disaggregation: put reading the prompt and writing the answer on separate GPUs, see Prefill and decode.
• Speculative decoding: Speculative decoding.
• Metrics: time to first token (TTFT), tokens per second per user, total throughput.
• Software: vLLM, SGLang, TensorRT-LLM, llama.cpp server, Ollama (simple, local).
• Costs: providers charge per million tokens, input and output separately.

See also: Tokens per second, Why GPUs for AI?