Initialisation, normalisation and dropout

In short: Three aids make deep training stable: clever starting values of the weights (initialisation), normalisation of intermediate values and dropout against memorising.

In more detail: Without them training of deep networks often fails because of exploding or vanishing values.

In detail

• Initialisation: not all weights to zero (otherwise all neurons learn the same). Random values with suitable spread: Xavier/Glorot for tanh/sigmoid, He/Kaiming for ReLU, so that values stay similarly large across many layers.
• Batch normalisation: normalises each value across the batch to mean 0 and spread 1, with learnable scaling. Common in image networks. Depends on batch size.
• Layer normalisation: normalises across the features of a single example. Standard in transformers. RMSNorm is a leaner variant in modern LLMs.
• Pre-norm versus post-norm: modern transformers normalise before the block, which stabilises training of very deep networks.
• Dropout: during training neurons are randomly switched off (for example 10%). The network cannot rely on single ones and overfits less. At use time dropout is off.
• Other regularisation: weight decay, data augmentation, early stopping.

See also: Overfitting and the bias–variance trade-off, Neurons, layers and activation functions