Self-attention in detail

In short: Self-attention lets every token decide which other tokens it wants to attend to and mixes their information accordingly. For this there are three vectors per token: query, key and value.

In more detail: A token’s query is compared with the keys of all tokens (dot product). The more similar, the higher the weight. Softmax turns this into shares, and the result is the weighted sum of the values.

In detail

Formula: Attention(Q,K,V) = softmax(Q·Kᵀ / √d) · V

1. Each token embedding is turned into Q, K, V by three weight matrices.
2. Q·Kᵀ gives a similarity for every pair of tokens (an n×n matrix).
3. Dividing by √d keeps the values in a good range.
4. Causal mask: in text generation a token may only look at earlier ones, later ones are set to −∞.
5. Softmax per row → weights that sum to 1.
6. Weights × V → new representation of the token that contains context.

• Illustrative example: in the sentence “The dog chased the cat because it was fast” the word “it” can fetch information from “cat” or “dog” via attention.
• Cost: the n×n matrix grows quadratically with text length. That is why long contexts are expensive, see Context window and FlashAttention.
• Note: attention has no built-in order information, it comes from Positional encoding and RoPE.

See also: Multi-head attention and variants, The transformer block in detail