Model compression: distillation and pruning

In short: Large models can be made smaller and faster: distillation (a small model learns from the big one), quantisation (fewer bits per number) and pruning (removing unimportant weights).

In more detail: The aim is to run on phones and small computers or to lower cost per request with as little quality loss as possible.

In detail

• Distillation: a “student” learns to imitate the outputs (probabilities) of a “teacher”. Soft probabilities contain more information than hard labels. Many small models arise this way.
• Quantisation: store weights in 8, 4 or fewer bits, see Quantisation and GGUF and llama.cpp.
• Pruning: remove unimportant weights or whole heads/layers. Structured pruning really speeds up, unstructured often only on special hardware.
• Low-rank decomposition: large matrices as a product of small ones, as in LoRA (Fine-tuning and LoRA).
• Speculative decoding: a small model guesses, the big one verifies, see Speculative decoding.
• Trade-off: less memory and faster, in return slightly worse, especially in arithmetic and rare knowledge.

See also: Quantisation, VRAM and model size