How a language model generates text: step by step

In short: A language model generates text token by token. For the next token it computes a probability for every word piece in the vocabulary, picks one, appends it and repeats until an end token comes.

In more detail: This is called autoregressive. It does not “plan” the whole sentence in advance, each step arises from the text so far.

In detail

1. Tokenise: text → token numbers (see Tokenisation: BPE and vocabulary).
2. Embedding: each number → vector (for example 4096 numbers).
3. Bring in positions (Positional encoding and RoPE).
4. Blocks: through all transformer blocks (The transformer block in detail).
5. Output: the last vector of the last position becomes logits (one number per token in the vocabulary) through a large matrix.
6. Softmax → probabilities (Softmax and probabilities).
7. Choose: greedily (highest probability), randomly by distribution (temperature, top-k, top-p), see Temperature and sampling.
8. Append and repeat. Thanks to the KV cache only the new token needs to be computed.

• Prefill and decode: reading the prompt (parallel, fast) is different from writing token by token (sequential, memory-limited), see Prefill and decode.
• Not stored: the model has no memory between conversations. Everything it “knows” is in the weights and the current context.
• Why it seems creative: random selection yields different continuations of the same question.

See also: How a language model is trained, Hallucination (AI)