Positional encoding and RoPE

In short: Attention by itself knows no order. Positional encoding gives each token information about where it stands. Widespread today is RoPE, which rotates vectors depending on position.

In more detail: Without position, “dog bites man” and “man bites dog” would be the same set of words for attention.

In detail

• Sinusoidal encoding: the original (2017) adds sine and cosine waves of different frequency to the embedding.
• Learned positions: one vector per position (GPT-2). Limited to the training length.
• RoPE (rotary position embedding): rotates Q and K pairwise by a position-dependent angle. The dot product then depends only on the distance of the tokens. Standard in Llama, Qwen, Mistral and others.
• ALiBi: subtracts a penalty proportional to distance from the attention scores.
• Length extension: models are trained for one length. With tricks (position interpolation, NTK, YaRN) RoPE is stretched to longer contexts, often with short additional training.
• Consequence: the usable context length is not only a memory question but depends on training, see Context window.

See also: Self-attention in detail, Context window