Cross-validation and data splitting: testing models fairly

In short: Training, validation and test data must stay separate; cross-validation uses the data several times in changing roles to estimate performance more reliably.

In more detail: A model measured on the training data set is a cheater: it has seen the answers. Only unknown data shows the true performance (generalisation).

In Depth

Three data sets

• Training (e.g. 70 %): the model learns parameters.
• Validation (15 %): choose hyperparameters, compare models, early stopping.
• Test (15 %): measure the final performance once at the end, never use it for tuning (otherwise “test data leakage”, optimistic numbers).
• Random shuffling (train_test_split, fix random_state) and stratification (keep class ratio).

k-fold cross-validation

Split the data into k parts (folds) (e.g. 5): k times one part is used for testing, the remaining k−1 for training; mean and spread of the results show performance and stability.

from sklearn.model_selection import cross_val_score, StratifiedKFold
scores = cross_val_score(model, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=0), scoring='f1')
print(scores.mean(), scores.std())

Advantages: all data is used, less dependence on chance; leave-one-out (k = n) with small data (expensive). Nested CV: outer loop for evaluation, inner for hyperparameter search (avoids optimism).

Special cases

• Time series: do not shuffle! The future must not enter training. TimeSeriesSplit (training = past, test = future).
• Groups (several rows per person/patient/device): GroupKFold, so the same person is not in training and test.
• Class imbalance: stratify, suitable metrics (F1, PR-AUC).
• Remove duplicates/near-duplicates before the split.
• Little data: cross-validation, bootstrapping, simpler models.

Typical mistakes (data leakage)

• Scaling/imputing/feature selection on the whole data set before the split → information flows from test into training. Solution: pipeline (make_pipeline) with preprocessing only on training folds.
• Target leakage: a feature contains the answer indirectly (e.g. “treatment given” in disease prediction).
• Iterating too often on the test set.
• Distribution shift (production ≠ training data): monitoring.
• Reporting: mean ± standard deviation, baseline (e.g. majority class), confidence.
• Good practice: baseline first, one clean holdout test, results reproducible (seeds, versions).

See also: Overfitting and underfitting, Metrics: precision, recall and F1, Hyperparameter search, Time series and forecasts