Overfitting and the bias-variance trade-off

In short: An overfitted model memorises the training data and fails on new data. A too-simple model learns too little (underfitting). The goal is the balance in between.

In more detail: A model’s error consists of bias (systematic simplification), variance (sensitivity to the specific training data) and noise. More complexity lowers bias but raises variance. What matters is performance on unseen data.

In Depth

• Splitting: Training, validation and test data. The test set is used only at the end.
• Cross-validation: Splitting several times for more stable estimates.
• Learning curves: Compare training and validation error. A large gap = overfitting.
• Remedies: More data, simpler models, regularisation (L1/L2, dropout), early stopping, data augmentation.
• Data leakage: Information from the test data gets into training and fakes good values.
• Metrics: Accuracy, precision, recall, F1, ROC-AUC; accuracy alone is misleading for imbalanced classes.
• Distribution shift: Real data differ from training.

See also: Gradient descent and learning from data, Neural networks and backpropagation