Benchmarks for language models

In short: Benchmarks are standardised tests used to compare language models, for example MMLU (general knowledge), HumanEval (code) or GSM8K (maths problems).

In more detail: They deliver comparable numbers but have limits: models may have seen test questions in training, and a high score says little about your own task. In addition there are rankings where people compare answers blind (for example Chatbot Arena).

In Depth

• MMLU: multiple-choice knowledge in many subjects.
• HumanEval, MBPP: programming tasks with automatic checking.
• GSM8K, MATH: maths and word problems.
• Contamination: If test data ends up in training, the score is too optimistic.
• Own tests: With 20 to 50 examples from your own work you can judge a model best.

See also: Parameters (AI model), Hugging Face