Quantisation

In short: Quantisation stores a model’s weights with fewer bits (for example 4 instead of 16). That saves memory and makes the model faster, but costs some quality.

In more detail: A weight in 16-bit floating point needs 2 bytes, in 8-bit 1 byte, in 4-bit half a byte. Since speed depends on memory bandwidth, a 4-bit model is roughly four times as fast as the same model in 16-bit.

Our context: In the AI speed calculator you can choose the quantisation or let it be determined automatically.

In Depth

• Common levels: Q8_0 (almost lossless), Q6_K, Q5_K_M, Q4_K_M (good compromise), Q3, Q2 (clear losses).
• Bits per weight: Q4_K_M is effectively about 4.8 bits because scale values are stored too.
• Quality loss: often small for large models, more noticeable for small ones. Below 3 bits quality suffers strongly.
• Formats: GGUF (llama.cpp), GPTQ, AWQ, EXL2, bitsandbytes, FP8.
• Training vs. running: Quantisation is usually done after training for operation.

See also: VRAM and model size, GGUF and llama.cpp, Number formats: FP32, FP16, BF16 and FP8