Decoding: greedy, beam search, top-k and top-p

In short: From the probabilities for the next token a choice must be made. The method for it is called decoding: greedy, beam search or random selection with top-k and top-p.

In more detail: The choice determines whether answers are safe and uniform or varied.

In detail

• Greedy: always the most probable token. Deterministic, tends to repetition.
• Beam search: follows the b best partial sentences in parallel and picks the best overall path at the end. Good for translation, usually flat and generic for chat.
• Random drawing: choose tokens by probability.
• Temperature: makes the distribution sharper (<1) or flatter (>1), see Temperature and sampling.
• Top-k: draw only from the k most probable tokens.
• Top-p (nucleus): draw from the smallest set whose probabilities add up to p (e.g. 0.9).
• Min-p: discards tokens below a fraction of the top probability.
• Repetition penalty: lowers already used tokens.
• Structured output: grammar constraints allow only valid JSON.
• Practice: facts and code: low temperature. Ideas and stories: higher.

See also: How a language model generates text: step by step, Softmax and probabilities