Multi-GPU and tensor parallelism

In short: With several graphics cards, a model that does not fit on a single one can be split.

In more detail: With a layer split (layer/pipeline) each card holds part of the layers and the cards work one after another. Memory adds up, but speed is about that of one card. With tensor parallelism all cards compute each layer simultaneously and constantly exchange intermediate results. That is faster but needs a fast link (NVLink or PCIe).

Our context: In the AI speed calculator you can select several cards. The calculator considers the split mode and the interconnect.

In Depth

• PCIe bandwidth: On consumer boards often x8 or x4 per card. That slows tensor parallelism.
• NVLink: fast direct link on workstation and data-centre cards, no longer on GeForce.
• Identical cards: Mixing different cards works; the slowest determines the speed.
• Power and cooling: Several cards need a power supply and case with space.

See also: VRAM and model size, Memory bandwidth