The transformer block in detail

In short: A transformer consists of many identical blocks. Each block has two parts: attention (exchanging information between tokens) and an MLP (letting each token “think” on its own), each with normalisation and a skip connection.

In more detail: A language model stacks for example 32 to over 100 such blocks. The knowledge sits mainly in the weights of the MLP parts.

In detail

Flow in a modern block (pre-norm):

1. x → RMSNorm → attention → + x (residual)
2. x → RMSNorm → MLP → + x (residual)

• MLP (feed-forward): two large matrices with an activation in between (SwiGLU). Hidden size usually ≈ 2.7 to 4 times the model width. Makes up about two thirds of the parameters.
• Attention: mixes information between positions, see Self-attention in detail.
• Model width (d_model): for example 4096 for a 7-billion model with 32 blocks.
• Input and output: at the start an embedding (token → vector), at the end a matrix that maps to logits for the vocabulary.
• Mixture of experts: instead of one MLP many experts, only a few active per token, see Mixture of experts (MoE).
• Counting parameters: roughly 12·layers·d² for dense models.

See also: Transformer architecture, Skip connections and residual networks