Evaluating AI: What Does “Good” Actually Mean?
A demo proves possibility. An evaluation estimates performance. Neither automatically proves usefulness in your environment.
A demo is not a trial
A demo proves possibility. An evaluation estimates performance under stated conditions. Neither automatically proves usefulness in your environment.
Those three objects get smashed together in headlines. A clip of one success becomes “it works.” A high score on a named test becomes “intelligence.” A score on last year’s test set becomes “ready for your clinic.” None of those leaps is free.
AI is a category of machine-based systems. [nist-ai-glossary] [oecd-ai-2024] It is not defined as learning. Search, rules, and planners can be evaluated too. See What is AI? and AI Is More Than Machine Learning.
This page is evaluation as an experiment: a task, a population, a scoring rule, a date, and a named system. It is not a redo of How to Check an AI Claim. That lesson is first-party versus rumor, model versus product, unpublished numbers. Use it when you inspect a named claim. Use this page when you ask what the measurement actually was.
One omelette is a demo. Fifty Tuesday omelettes are an evaluation.
A cooking video of one perfect omelette is a demo. It shows that a good omelette is possible, under that kitchen, that pan, that morning.
Tasting 50 omelettes on a Tuesday night shift is an evaluation. You named a task (omelettes, that menu). You named a population (Tuesday night, that line). You named a scoring rule (did the ticket go out, was it overcooked, did the table send it back). You named a date. You named a system (that cook, that station, that recipe).
Your kitchen, your pan, your eggs are deployment. The estimate from Tuesday does not automatically transfer. Different eggs. Different heat. Different rush. Different cook.
A cherry-picked clip of the one perfect omelette is not the Tuesday trial. A high score on someone else’s test is not a proof that the thing is useful in your kitchen.
What an evaluation needs
Name five things, or you do not have an evaluation. You have a vibe.
- A task. What decision or output is being scored? Spam or ham. Translate this sentence. Draft this note. Not “be intelligent.”
- A population. Which cases, from which process, in which setting? Tuesday night tickets. Last year’s clinic. Public web questions from a particular year.
- A scoring rule. How is a case counted as success or failure? Accuracy is one rule. Precision and recall are others. Human preference is another family. The rule has to match the decision.
- A date. When was the run? Products change. Test sets leak. A score from last spring is a score from last spring.
- A system. A model is a component. A product wraps a model with tools, retrieval, filters, and a user interface. An agent may call tools. The evaluation is of the thing that actually ran. How to Check an AI Claim already taught model versus product. If the system uses tools, you are scoring the composed system, not the base model alone.
Data: What Machines Actually Learn From already said the dataset is a sample, and that a train/test leak is a fake check. This page goes further: leakage is an experimental-design failure. The estimate is not merely noisy. It is answering the wrong question.
Accuracy is one metric. It can hide the miss that matters.
Accuracy is the fraction of cases the system got right, under that scoring rule, on that sample. It is one number. It is not a universal “good.”
Rare classes break the story. If almost every message is ordinary mail, a filter that never catches spam can still look accurate. It missed the rare class. The number hid the miss.
Precision and recall answer different questions. Illustration, not a study. A spam filter:
- Recall is whether you catch the spam. Miss a real spam, recall drops.
- Precision is whether the things you called spam really were spam. Accuse ordinary mail (ham), precision drops.
You cannot invent one official cutoff that makes a system “good.” The metric has to match the decision cost. Missing spam is one cost. Accusing ham is another. Those are not the same question.
A high accuracy is still not a calibrated probability of being right in your clinic. [guo2017] Probability, Uncertainty, and Confidence already taught that 0.85 is a score until someone shows it is calibrated. This page does not redo that lesson.
A benchmark is a named test, not intelligence
A benchmark is a dataset, plus a scoring rule, plus a protocol: how you run the system, what you are allowed to peek at, when the test is used. ImageNet is a particular vision benchmark built that way, not a theory of seeing. [deng-imagenet-2009] HELM (Holistic Evaluation of Language Models) is built around many scenarios and many metrics, not one headline number. [liang2022helm]
A high score on that test, under those conditions, at that time, is a high score on that test. It is not general intelligence. It is not a proof that the system will help in your environment.
Three ways the estimate goes fake or stale:
- Contamination. Test items, or close copies, were in training. Brown, Mann, Ryder, and coauthors discussed data contamination as a methodological issue for models trained on large web corpora. [brown2020gpt3] HELM discusses train-test contamination as a limit on what a score means. [liang2022helm] OpenAI’s GPT-4 System Card is a first-party writeup of the evaluations that lab chose to run and disclose. [openai-gpt4-system-card] This page does not invent a contamination rate.
- Leaderboard overfitting. Many people tune until they win that board. The board stops being an independent check. It becomes another training signal.
- Stale scores. The product changed after the run. Tools were added. The prompt wrapper changed. Yesterday’s number is yesterday’s system.
A cherry-picked clip is not a trial. Possibility is not typical performance.
People picking A or B is a preference test
A human preference test asks people to pick output A or output B under some instructions. Ouyang, Wu, Jiang, and coauthors trained InstructGPT with rankings of that kind: people compared model outputs, and those comparisons became the training signal. [ouyang2022instructgpt]
That measurement is real. It is also narrow. It measures preference under those instructions, for those raters, on those prompts. It is not “truth.” It is not “your users.” It is another metric family.
The evaluation population is not your clinic
Data: What Machines Actually Learn From already said the sample is not the world. Here the point is transfer. The estimate was computed on that population, that year, that protocol. Your clinic, your city, your year can be a different process. Dataset shift is the name for that change. [quinonero-candela-2009] The number does not transfer automatically.
What to do as a reader
- Name the task, the population, the metric, the date, and the system (model, product, or agent).
- Ask whether test items, or close copies, were in training.
- Do not treat a demo as a trial. One success proves possibility.
- Do not treat a benchmark score as general intelligence.
- Do not invent a universal “good” number. Match the metric to the decision.
- Related lessons: How to Check an AI Claim for a named-system claim. Probability, Uncertainty, and Confidence for scores versus calibrated probabilities. Data: What Machines Actually Learn From for sample versus world. Machine Learning Fundamentals for training. Tokens, Embeddings, and Context for how language-model inputs are built. What is AI? for the category. AI Is More Than Machine Learning for methods that are not this pipeline.
References
National Institute of Standards and Technology (NIST) (2026). Artificial Intelligence — Glossary. NIST Computer Security Resource Center.
National Institute of Standards and Technology (NIST) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1.
Organisation for Economic Co-operation and Development (OECD) (2024). Explanatory memorandum on the updated OECD definition of an AI system. OECD Artificial Intelligence Papers.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.
Liang, P., Bommasani, R., Lee, T., et al. (2022). Holistic Evaluation of Language Models. arXiv preprint.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). ImageNet: A Large-Scale Hierarchical Image Database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D. (Eds.) (2009). Dataset Shift in Machine Learning. MIT Press.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS).
OpenAI (2023). GPT-4 System Card. OpenAI.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS).
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL).
Pineau, J., Vincent-Lamarre, P., Sinha, K., et al. (2021). Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arXiv preprint.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR 70.
Citation Note: Some citations are open access (arXiv or DOI, where linked). Others are books or journal articles that may sit behind a publisher paywall. Use the linked DOI or arXiv when available. If you notice any citation errors, please let us know.