Transformer architecture

In short: The transformer is the architecture behind almost all modern language models. Its core idea is attention.

In more detail: In the attention mechanism the model computes, for each token, how important every other token of the context is to its understanding. This allows distant relationships to be captured. The architecture was published in 2017 in the paper “Attention Is All You Need”.

In Depth

• Layers: many identical blocks of attention and a feed-forward network.
• Decoder-only: GPT, Llama, Qwen generate text token by token.
• Quadratic growth: The attention computation grows quadratically with context length. FlashAttention and other methods ease that.
• Positions: rotary embeddings (RoPE) encode the order.

See also: Parameters (AI model), KV cache