Knowledge distillation: Small models learn from large ones

In short: In distillation a small “student” model learns to imitate the outputs of a large “teacher” model and thereby becomes much faster at similar quality.

In more detail: Large language models are expensive to run. Distillation transfers their knowledge to a smaller model. Instead of training only on correct answers, the student sees the teacher’s probability distribution and thereby learns more than from hard labels.

In Depth

The idea

In image recognition a teacher might report for a cat photo: 90 % cat, 8 % lynx, 2 % dog. These soft targets reveal that cat and lynx are similar. A student that only sees “cat” as the correct answer does not get this information. Geoffrey Hinton and colleagues described in 2015 how to smooth such probabilities with a temperature and use them as a training target.

With language models

The student is trained on texts that the teacher generated (sequence distillation), or directly on the teacher’s probabilities for the next token (logit distillation). Current small models such as some variants of Llama, Gemma or DistilBERT use such methods. Answers from a strong model with chains of reasoning are also used as training data for smaller models.

Benefits

• Faster and cheaper to run, less memory.
• Runs on phone or laptop (edge AI).
• Often markedly better than a model of the same size trained only on original data. Limits: the student rarely surpasses the teacher, adopts its errors and biases and often loses rare abilities.

Legal matters and practice

Some providers’ terms of use forbid using outputs of their models to train competing models. Open teacher models can usually be used more freely. Distillation complements quantisation and pruning, which shrink a finished model instead of training a new one.

See also: Quantisation, Fine-tuning and LoRA, Edge AI, Temperature and sampling