Why GPUs for AI?

In short: Neural networks consist of huge matrix multiplications that break down into thousands of identical, independent calculations. Graphics cards have thousands of small compute cores and are built for exactly that.

In more detail: A CPU has a few strong cores for varied tasks. A GPU has thousands of simple cores that execute the same operation on many data at once (SIMD/SIMT).

In detail

• Tensor cores: special units for matrix blocks at low precision (FP16, BF16, FP8, FP4). They deliver most of the AI compute.
• Memory bandwidth: when generating text the whole model must be read from memory for every token. Bandwidth therefore limits speed, see Memory bandwidth.
• HBM: high-bandwidth memory right at the chip, several TB/s.
• CUDA: NVIDIA’s programming platform, largest ecosystem. Alternatives: ROCm (AMD), Metal (Apple), SYCL/oneAPI (Intel), Vulkan.
• TPU, NPU: special AI chips, see NPU and AI accelerators.
• Number formats: lower precision = more throughput, see Number formats FP32, FP16, BF16 and FP8.
• Why not CPU: possible, but 10 to 100 times slower for large models.
• Compute: stated in FLOPS (operations per second), modern data-centre GPUs reach petaflops at low precision.

See also: VRAM and model size, Training at scale: GPUs, parallelism and memory