How do we measure whether AI is any good?
Quality has to be measured for a specific task, group of users and set of conditions.
What is it?
AI quality is measured with tests (benchmarks), but each test measures only one task under particular conditions. A model that excels on one leaderboard can be average at your task, in your language or with your data.
An everyday example
A model tops the leaderboard for maths problems in English but makes mistakes summarising Slovenian contracts.
Why does it matter?
Leaderboards influence purchases and headlines. Understanding their limits keeps you from falling for claims about “the best model”.
A common misconception
That there is a single “best model”. Without a stated task, language and conditions, such a ranking is not useful.
In more detail
What is a benchmark?
A shared set of tasks or tests used to compare systems.
Why is one ranking not enough?
A model can be good at maths, weaker at perception, different in a smaller language such as Slovenian, and unreliable when using tools.
What is test contamination?
When a model has already seen the test tasks, or very similar ones, during training, so its score can mislead.
What should I measure in practice?
Accuracy, error rate, safety, speed, cost, language, accessibility and the consequences of a wrong answer.
Remember: “The best model”, without saying for which task, is not a useful verdict.
Reliable sources
Last reviewed 1 November 2026. An explainer, not professional, legal or medical advice. For rules and figures that change, check the official version.