EMZETT.VRAM and model size
In short: VRAM is the memory of the graphics card. A language model runs fastest when it fits completely.
In more detail: The memory need in gigabytes is roughly: parameters (in billions) × bits per weight ÷ 8, plus 10 to 20 % for context and intermediate values. A 70-billion model needs 140 GB in 16-bit and about 40 GB in 4-bit.
Our context: The AI speed calculator checks whether model and context fit into VRAM. Otherwise the remainder is offloaded to the slower main memory (offloading).
In Depth
| Model | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 8 bn | 16 GB | 8 GB | 4.5 GB |
| 32 bn | 64 GB | 32 GB | 18 GB |
| 70 bn | 140 GB | 70 GB | 40 GB |
- KV cache: grows with context length and adds to the model (KV cache).
- Several cards: The memory adds up; speed depends on the split (multi-GPU and tensor parallelism).
- Rule of thumb for buyers: 24 GB of VRAM allow models up to about 30 billion parameters in 4-bit, 48 GB up to about 70 billion.
See also: Quantisation, Memory bandwidth