Forward and backward pass in detail

In short: In the forward pass the data travels layer by layer through the network and yields the prediction. In the backward pass the chain rule computes how each weight influenced the error.

In more detail: The backward pass (backpropagation) is efficient because it reuses intermediate results of the forward pass. Frameworks like PyTorch do this automatically (“autograd”).

In detail

1. Forward: x → layer 1 → layer 2 → … → prediction ŷ. Intermediate values (activations) are stored.
2. Loss: L = loss(ŷ, y).
3. Backward: from back to front: ∂L/∂W for each layer using the chain rule: ∂L/∂x = ∂L/∂y · ∂y/∂x.
4. Update: the optimiser changes weights, see Optimisers: SGD, momentum and Adam.
5. Zero the gradients, next batch.

• Memory: the stored activations are a large part of training memory. Gradient checkpointing stores only a few and recomputes the rest (saves memory, costs time).
• Compute: backward is about twice as expensive as forward. Training ≈ 3× inference per token.
• Vanishing/exploding gradients: with many layers gradients become tiny or huge. Remedies: ReLU/GELU, normalisation, skip connections, gradient clipping.
• Code (PyTorch): loss.backward() computes, optimizer.step() updates.

See also: Neural networks and backpropagation, Skip connections and residual networks