Memory bandwidth

In short: Memory bandwidth indicates how many gigabytes per second can be transferred between memory and processor.

In more detail: It is the decisive figure for the speed of language models. Graphics cards have hundreds to thousands of GB/s (GDDR6X, HBM), while normal PC memory only has about 50 to 100 GB/s. Apple chips with unified memory lie in between (100 to 800 GB/s).

Our context: In the AI speed calculator every graphics card, processor and Apple chip shows its bandwidth. It determines how many tokens per second are possible.

In Depth

• Calculation: bus width in bits ÷ 8 × memory data rate. A 384-bit card with 21 Gbit/s per pin has 384 ÷ 8 × 21 = 1,008 GB/s.
• Dual channel: Two RAM sticks double the bandwidth compared with one. DDR5-6000 delivers about 96 GB/s in dual channel.
• HBM: stacked memory in data-centre cards (terabytes per second).
• Utilisation: In practice you often reach 60 to 85 % of the maximum.

See also: Tokens per second, VRAM and model size, Unified memory (Apple Silicon)