Number formats: FP32, FP16, BF16 and FP8

In short: AI computes with floating-point numbers of various precision. Fewer bits save memory and compute time but introduce rounding errors.

In more detail: FP32 (32-bit) is the classic precision. FP16 and BF16 (16-bit) are standard for training and inference. BF16 has the value range of FP32 with lower precision and is more stable in training. FP8 (8-bit) is used by newer cards (Hopper, Ada, Blackwell) for even more speed.

In Depth

• Structure: sign, exponent (value range), mantissa (precision). BF16: 8-bit exponent, 7-bit mantissa. FP16: 5-bit exponent, 10-bit mantissa.
• Tensor cores: special units of the graphics card for matrix maths in FP16, BF16, FP8 and INT8.
• Integer formats: INT8 and INT4 are even smaller and mainly intended for inference.
• TFLOPS: measure of compute. When generating it matters less than bandwidth; when processing long prompts it matters a lot.

See also: Quantisation, Prefill and decode