NPU and AI accelerators

In short: An NPU (neural processing unit) is a chip area that runs neural-network computations especially efficiently. It is in phones and new laptop processors.

In more detail: NPUs are designed for low power consumption and quote their performance in TOPS (trillions of operations per second, usually INT8). They are ideal for small models, camera effects and background tasks. For large language models, memory bandwidth is usually the bottleneck, not the TOPS figure.

In Depth

• Examples: Apple Neural Engine, Qualcomm Hexagon, Intel/AMD NPUs, Google TPU (in data centres).
• Copilot+ PCs: require at least 40 TOPS.
• GPU vs. NPU: GPUs are more flexible and faster, NPUs more frugal.
• Marketing numbers: TOPS figures are only comparable at the same precision.

See also: Memory bandwidth, Number formats: FP32, FP16, BF16 and FP8