Offloading models

In short: If a model does not fit in VRAM, part of it can be offloaded to main memory (RAM). It works but is slower.

In more detail: Two methods: with a layer split the first layers stay on the graphics card and the rest run on the processor. With tensor offload the compute-intensive parts stay on the card, the large, rarely used expert weights in RAM. The latter is very effective with MoE models.

Our context: The AI speed calculator calculates both methods and shows how much is offloaded and the resulting speed.

In Depth

• Why slower: Total time is made of the VRAM part and the RAM part. The RAM part at 50 to 100 GB/s quickly dominates.
• PCIe: The link between card and RAM matters with offload.
• In practice: Offloading a few layers costs little. Offloading half loses a lot of speed.
• Alternative: smaller model or stronger quantisation.

See also: VRAM and model size, Memory bandwidth