Overfitting and underfitting: When models memorise

In short: A model with overfitting knows its training data perfectly but fails on new data; underfitting means it does not learn even the training data well.

In more detail: The goal of machine learning is generalisation: a model should do well on data it has never seen. There are two typical problems: it learns too little (underfitting) or too much by rote (overfitting).

In Depth

A picture of it

Imagine learning for an exam by memorising only the solutions of the practice tasks. On the practice tasks you are perfect (small training error); on new tasks you fail (large test error). That is overfitting. Whoever instead learns only superficially and cannot even solve the practice tasks has underfitting (high error everywhere).

Recognising it

Data is split into training, validation and test. If the error on the validation data rises while the training error keeps falling, overfitting begins. The learning curves show this: with underfitting both curves are high, with a good fit both are low and close together, with overfitting there is a large gap.

Countermeasures against overfitting

• More (and more varied) data, data augmentation.
• A simpler model, fewer parameters.
• Regularisation (L1/L2, weight decay), which penalises large weights.
• Dropout: randomly switch off neurons.
• Early stopping: stop training when the validation error rises.
• Cross-validation for more reliable testing.

Countermeasures against underfitting

A more expressive model (more layers, more parameters), better features, longer training, less regularisation. Between the two extremes lies the bias-variance trade-off: too simple models have high bias (systematic errors), too complex ones high variance (too sensitive to chance in the training data). For very large language models overfitting is less of a problem as long as the data is huge and varied, but memorisation of training data can still occur (data protection, copyright).

See also: Batch, epoch and learning rate, Data, features and labels, Gradient descent