Softmax and probabilities

In short: Softmax turns a list of arbitrary numbers (logits) into probabilities that are all positive and sum to 1. It is the last stage of almost every classifier and language model.

In more detail: Formula: softmax(zᵢ) = e^zᵢ / Σ e^zⱼ. Larger inputs get distinctly larger shares because the exponential function amplifies differences.

In detail

• Example: logits (2.0, 1.0, 0.1) → e-values (7.39, 2.72, 1.11) → sum 11.22 → probabilities (0.66, 0.24, 0.10).
• Temperature: if you divide the logits by T before softmax, the distribution gets sharper (safer) at small T and flatter (more creative) at large T, see Temperature and sampling.
• Numerical stability: subtract the maximum before computing so that e^z does not overflow.
• In language models: one value per token of the vocabulary (for example 100,000). The next token is drawn from it.
• Sigmoid: the special case for two classes.
• Attention: there too softmax distributes the weights over the positions, see Self-attention in detail.

See also: Loss functions: MSE and cross-entropy, Token (AI)