Metrics: accuracy, precision, recall and F1

In short: Metrics measure how good a model is. For classification these are accuracy, precision, recall and F1. Which one counts depends on which errors are worse.

In more detail: The basis is the confusion matrix with four cells: true positive (TP), false positive (FP), true negative (TN), false negative (FN).

In detail

• Accuracy: (TP + TN) / all. Misleading with imbalanced classes: if only 1% are sick, “always healthy” reaches 99%.
• Precision: TP / (TP + FP). Of everything reported positive, how much is right? Important when false alarms are costly (spam filter deletes real mail).
• Recall (sensitivity): TP / (TP + FN). Of everything positive, how much was found? Important when missing is costly (cancer screening).
• F1: harmonic mean of precision and recall: 2·P·R / (P + R).
• ROC-AUC: area under the curve of hit rate versus false alarm rate, independent of the threshold.
• Regression: MAE (mean absolute error), RMSE, R².
• Language models: perplexity, benchmarks, human evaluation, see Benchmarks for language models.
• Threshold: the threshold for “positive” is selectable and shifts precision against recall.

See also: Training, validation and test, Classification and regression