Training, validation and test

In short: Data is split into three parts: training to learn, validation to tune the settings (hyperparameters) and test for the honest final evaluation, which is looked at only once.

In more detail: If you measured only on the training data you would reward memorising. The separation shows how well the model really generalises.

In detail

• Hyperparameters: settings chosen before training (learning rate, network size, number of layers). Parameters, in contrast, are learned.
• Cross-validation: split data into k parts, use each part once for validation, k times, and average. Good with little data.
• Early stopping: stop training when validation error rises again.
• Time series: do not shuffle, split by time (training earlier, test later).
• Avoid leakage: compute preprocessing (e.g. scaling) on training data only.
• Benchmarks: if public test questions are on the internet, models may have seen them in training (contamination), see Benchmarks for language models.

See also: Metrics: accuracy, precision, recall and F1, Overfitting and the bias–variance trade-off