Confusion matrix, ROC and AUC: evaluating classifiers precisely

In short: The confusion matrix counts correct and wrong predictions per class; the ROC curve and AUC show how well a model separates across all thresholds.

In more detail: “95 % accuracy” sounds good but can be worthless: if 95 % of mails are not spam, a model that always says “not spam” also reaches 95 %. That is why more detailed measures are needed.

In Depth

Confusion matrix

For yes/no (positive/negative):

	predicted +	predicted −
actually +	TP (true positive)	FN (missed)
actually −	FP (false alarm)	TN (true negative)
Example spam filter, 1000 mails (100 spam): TP = 80, FN = 20, FP = 10, TN = 890.

• Accuracy = (TP+TN)/all = 970/1000 = 0.97.
• Precision = TP/(TP+FP) = 80/90 ≈ 0.89 (how many “spam” hits are right?).
• Recall/sensitivity = TP/(TP+FN) = 80/100 = 0.80 (how much spam is found?).
• Specificity = TN/(TN+FP) = 890/900 ≈ 0.99.
• F1 = 2·P·R/(P+R) ≈ 0.84.

Threshold and costs

Models often output probabilities; only a threshold (e.g. 0.5) turns them into “yes/no”. Lower threshold → more hits (recall ↑), more false alarms (precision ↓). What is right depends on the costs: cancer screening wants high recall (overlook nothing), a spam filter for important mail high precision (lose no real mail), fraud detection weighs both. Multi-class: N×N matrix, macro/micro average of metrics.

ROC and AUC

The ROC curve plots for all thresholds the true-positive rate (recall) against the false-positive rate (1 − specificity). Diagonal = guessing (AUC 0.5); the further top left, the better. AUC (area under the curve) = probability that a random positive case is ranked higher than a random negative case: 1.0 perfect, 0.5 chance.

from sklearn.metrics import confusion_matrix, roc_auc_score, precision_recall_curve
print(confusion_matrix(y_test, y_pred))
print(roc_auc_score(y_test, probabilities))

Which measure when?

• Balanced classes: accuracy, ROC-AUC.
• Rare positive class (fraud, disease): precision, recall, F1, PR-AUC (precision-recall curve); ROC looks too optimistic there.
• Regression: MAE, RMSE, R².
• Ranking/search: MAP, nDCG; language: perplexity, BLEU/ROUGE, human evaluation.
• Fairness: compare metrics per group (equality of error rates).
• Calibration: do probabilities hold (80 % → 80 % hits)?
• Good practice: look at the matrix, read error examples, compare a baseline, fix costs and choose the threshold accordingly.

See also: Metrics: precision, recall and F1, Classification and regression, Cross-validation, AI evaluation