Regularisation and early stopping: Curbing overfitting

In short: Regularisation penalises overly complex models, and early stopping ends training before the model memorises training data; both improve performance on new data.

In more detail: A model with very many parameters can reproduce the training data almost perfectly and learn the noise along. On new data it then performs worse (overfitting). Regularisation methods keep the model simple enough.

In Depth

L2 and L1 regularisation

A penalty term for large weights is added to the loss function:

• L2 (ridge, weight decay): sum of the squares of the weights. Pulls all weights towards zero but rarely exactly to zero. Standard in neural networks (weight_decay).
• L1 (lasso): sum of the absolute values. Sets unimportant weights exactly to zero and so selects features.
• Elastic net: mix of both. A parameter (λ or alpha) controls strength; it is chosen by cross-validation.

Dropout and data augmentation

• Dropout: during training neurons are switched off at random; the network cannot rely on individual ones.
• Data augmentation: enlarge training data artificially (rotate, flip images, noise).
• Fewer parameters or smaller networks, collect more data.
• Ensemble methods average several models.

Early stopping

You watch the validation error during training. If it no longer falls or rises again while training error keeps falling, training is stopped and the best model kept:

# Keras
EarlyStopping(monitor="val_loss", patience=5, restore_best_weights=True)

patience is the number of epochs without improvement that you wait. Simple and very effective.

Diagnosis

Learning curves show it: low training error, high validation error means overfitting (more regularisation, more data). Both high means underfitting (more complex model, better features, longer training). Never test on data you used to choose the regularisation strength; for that keep an untouched test set.

See also: Overfitting and bias-variance, Initialisation, normalisation and dropout, Training, validation and test, Loss functions