Multi-head attention and variants

In short: Instead of one attention there are several “heads” that attend in parallel to different relationships (grammar, references, proximity). Their results are combined.

In more detail: The vector is split into parts, each head has its own Q, K, V weights and learns a different kind of relation.

In detail

• Multi-head (MHA): for example 32 heads, each with 128 dimensions. Outputs are concatenated and mixed linearly.
• What heads learn: some track the previous token, some brackets, some names or repetitions (“induction heads” copy patterns).
• Multi-query (MQA): all heads share the same K and V. Saves a lot of KV-cache memory.
• Grouped-query (GQA): compromise: groups of heads share K/V. Standard in many models (Llama, Mistral).
• MLA (multi-head latent attention): compresses K/V into a small latent vector (DeepSeek).
• Importance: the KV cache grows with the number of K/V heads, GQA/MQA/MLA reduce it strongly.
• Sliding window and sparse attention: each token sees only a window or selected positions, saving compute for long texts.

See also: Self-attention in detail, KV cache