Skip connections and residual networks

In short: A skip connection (residual connection) adds the input of a block to its output: y = x + f(x). This lets gradients flow unhindered through very deep networks.

In more detail: The method comes from ResNet (2015) and made image networks with over 100 layers trainable. Every transformer block uses the same idea.

In detail

• Idea: the block only has to learn the deviation (the “residual”) from the input. “Doing nothing” is easy (f = 0).
• Gradient flow: in the backward pass there is a direct path without shrinking.
• Residual stream: in transformers the sum of all block contributions forms a “stream” per position that attention and MLP read and write.
• Variants: DenseNet (all earlier layers), U-Net (skips between encoder and decoder, in diffusion models).
• Importance: without skip connections networks with hundreds of layers would be practically untrainable.

See also: Forward and backward pass in detail, The transformer block in detail