Probability, entropy and KL divergence

In short: Language models are probability models. Entropy measures how uncertain a distribution is, KL divergence how strongly two distributions differ. Both appear in loss, sampling and fine-tuning.

In more detail: A probability distribution assigns each possibility a value between 0 and 1, the sum is 1.

In detail

• Conditional probability: P(next token | text so far). That is exactly what a language model computes. The probability of a whole sentence is the product of these factors (chain rule).
• Entropy: H = −Σ p·log p. High with evenly spread uncertainty, zero with certainty. It is the average “surprise” in bits.
• Cross-entropy: average surprise when guessing with distribution q while the truth is p. That is the training loss, see Loss functions: MSE and cross-entropy.
• KL divergence: KL(p‖q) = Σ p·log(p/q). Always ≥ 0, zero only for equal distributions. In RLHF it prevents the model from drifting too far from the starting model, see RLHF and preference optimisation.
• Perplexity: 2^entropy, the effective number of equally likely alternatives.
• Bayes: P(A|B) = P(B|A)·P(A) / P(B). Basis of naive Bayes spam filters.
• Random drawing: a token is drawn at random from a distribution, see Temperature and sampling.

See also: Softmax and probabilities, Maths basics for AI: vectors, matrices, tensors