Reading learning curves and diagnosing training

In short: Learning curves show training and validation error over time or over data size and reveal whether a model is over- or underfitted and what might help next.

In more detail: When a model is bad, guessing helps little. Learning curves provide a diagnosis. You plot the loss or accuracy for training data and validation data over the epochs (or over increasing training set size).

In Depth

Typical patterns

• Both curves fall and converge: healthy training.
• Training error low, validation error clearly higher or rising: overfitting. Remedies: regularisation, dropout, early stopping, more or augmented data, smaller model.
• Both high: underfitting. Remedies: larger model, better features, longer training, less regularisation, check a higher learning rate.
• Loss jumps wildly or becomes NaN: learning rate too high, unstable data, faulty normalisation.
• Loss hardly falls: learning rate too small, error in code or labels, dead neurons.

Curves over data size

If you plot the error against the size of the training set, you see: if training and validation converge at a high error level, more data does not help, but a better model or features do. If a large gap remains, more data or stronger regularisation usually help.

Practical steps

1. Baseline and small subset (can the model memorise 100 examples? If not, there is a bug.)
2. Sample-check data and labels.
3. Watch curves in TensorBoard, Weights & Biases or matplotlib.
4. Test one change at a time, fix random seeds.
5. Check on the test set only at the end.

Further indicators

Besides loss: accuracy, precision/recall, F1, for language models perplexity, for images IoU. Also look at gradient magnitudes and weight distributions, at learning-rate schedules (warm-up, cosine) and at differences between groups (fairness). A good curve does not replace checking the predictions on content.

See also: Overfitting and bias-variance, Regularisation and early stopping, Gradient descent, Metrics