AI chips compared: GPU, TPU, NPU and more

In short: AI mostly computes matrix multiplications. Special chips do many of them at once and efficiently: GPU, TPU, NPU and other accelerators.

The main types

• CPU: few, very flexible cores. Good for control, bad for huge matrices.
• GPU: thousands of small cores, originally for graphics, see Why GPUs for AI. Today standard for training and running. Special tensor cores speed up matrix maths at low precision.
• TPU (Google): chips tailored to matrix maths with so-called systolic arrays, linked in large clusters.
• NPU: a small, power-saving AI unit in phones and laptops, for example for photo and speech recognition on the device.
• Special chips (Groq, Cerebras and others): own architectures, for example far faster on-chip memory or a whole wafer as one chip to save data traffic.

What really counts

1. Compute in FLOPS at the precision used (FP16, FP8, INT4).
2. Memory and memory bandwidth: during text generation the chip mostly waits for data, not for computing, see Inference servers.
3. Links between chips when a model is spread over many (Training at scale).
4. Software: drivers and libraries often decide how much performance arrives.
5. Energy and cost.

See also: Edge AI, Model shrinking