Why Evals Matter

"To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of." — Ronald Fisher, address to the Indian Statistical Congress (1938)

Fisher was making a point about experimental design: the analysis cannot rescue an experiment that was not set up to answer the question. The AI-engineering version is that no amount of investigation after the fact will tell you whether last month's prompt change helped, because nothing was measured when it went in. You have a system that behaves differently than it did, a set of changes any of which might be responsible, and no way to attribute the difference to any of them.

Every part of this book has deferred to this one. Part III built an eval harness for prompts and admitted it was the only chapter whose content would survive the models changing. Part IV built one for retrieval and spent most of its final chapter on the ways a RAG system fools you. Part V introduced agents, where a single run touches a dozen decisions and the failure could be in any of them. Each time the answer was the same: measure the stages separately, against fixed cases, and stop reasoning about whether it worked.

This part generalises that into a discipline, and it is worth being direct about the stakes. Evaluation is the thing that makes AI engineering engineering. Without it you have a practice that looks like software development — code, reviews, deploys — wrapped around a component whose behaviour nobody can characterise and whose changes nobody can attribute. That is not a slower version of engineering. It is a different activity that resembles it.

The difficulty is real, though, and this chapter takes it seriously rather than scolding. Testing a model is genuinely not like testing a function, for four specific reasons, and every team that skips evals has usually run into at least one of them and concluded the problem was intractable. It is not. But the standard testing toolkit does not transfer, and knowing exactly which part fails to transfer is the difference between building something useful and building an elaborate way to feel measured.

Four reasons this is not unit testing

The obstacles are specific. Naming them is most of the work, because each has a known response.

The third test is the one that stops teams, so it deserves the emphasis. For open-ended output there is no answer key, and the response is to stop looking for one. You are not going to compare a generated summary against The Correct Summary. You are going to check properties: does it contain the required facts, does it cite its sources, is it within the length bound, does it avoid the forbidden claims. Those are computable, and together they are a usable definition of "good enough".

What an eval set does that nothing else can

Four jobs, and the fourth is the one people do not expect.

The fourth job is worth sitting with. An eval set is a product specification that happens to be executable. Writing the cases is the point at which a team discovers it disagrees about what the feature should do — whether it should refuse when unsure, how long an answer should be, whether a plausible-but-uncited claim is acceptable. That disagreement existed before anyone wrote a case; the eval set is simply where it surfaces, which is much better than launch.

The maturity ladder

Most teams are at level one and describe themselves as level three. The ladder is worth stating plainly so you can locate yourself honestly.

The fifth test names the pattern I see most often: a fixed set exists, someone runs it manually before big releases, one sample per case, nobody blocks on the result. That is level one being described as level three, and the gap matters because the practices differ in what they can actually catch.

The cost argument, which is not close

The objection to building evals is almost never that they are a bad idea. It is that they feel like overhead. The arithmetic does not support that.

The last test carries the argument I would lead with in any real conversation. The eval set is the only artefact in an AI system that appreciates. Prompts get rewritten, models get replaced, the framework gets swapped for whatever is fashionable next year — and a corpus of real cases with known-good answers, grown from your own production incidents, is worth more every month it exists.

Goodhart, and the honest caveat

A chapter arguing for measurement owes you the failure mode of measurement.

The second test names the cheapest defence available: a holdout set you never tune against. If accuracy on the tuning set climbs while holdout accuracy stays flat, the prompt did not get better — it got better at those cases.

And the sixth test states the position plainly, because Goodhart's law is frequently recruited as an argument for not measuring at all. Measuring means you will partly optimise the measure, which is a problem you manage with holdouts, metric diversity and periodic human review. Not measuring means you cannot tell whether you are improving, which is not a problem you manage — it is a condition you live in.

Common Mistakes

• Concluding evaluation is impossible after hitting one obstacle. Non-determinism, no oracle, unbounded inputs and a moving target are four separate problems, each with its own response.

• Looking for ground truth on open-ended output. There is no correct summary. Score computable properties — required facts present, sources cited, length bound respected, forbidden claims absent.

• Aiming for coverage. The input space is all of natural language. Sample real traffic and let production failures write your cases.

• Assuming a green suite means an unchanged system. It means the system was as you left it, on the model version the suite last ran against.

• Skipping the hour of reading raw outputs. Twenty real inputs, run and read side by side, tells you what to write cases about. Designing an eval set without it measures imagined usage faithfully for a year.

• Letting the people who built it be the only reviewers. Engineers grade their own system's output on effort — the work is visible and impressive. Someone who sees only the answer grades the answer.

• Treating an eval set as purely technical. It is an executable product specification, and writing it is where a team discovers it disagrees about what "good" means.

• Describing level one as level three. A fixed set, run manually before releases, one sample per case, nobody blocking on it — that is level one, and the distinction determines what it can catch.

• Measuring without gating. A suite nobody blocks on is a dashboard. The step from measuring to gating is organisational, and it is where most harnesses stall.

• Designing level four before building level one. Twenty cases and a crude metric is an afternoon. No rung is expensive given the one below it.

• Claiming cost as the blocker. A serious run is around eleven dollars; a small one is sixty-five cents. When a team says evals are too expensive, cost is not the reason.

• One headline metric. Accuracy up, groundedness down, latency up, cost up — a single number calls that a win. Report the vector.

• No holdout. Tuning against every case you have means you cannot distinguish "better" from "better at these cases".

• A corpus that never changes, or one that changes constantly. Grow steadily from incidents and version it, so old comparisons stay interpretable.

• Dropping human review once metrics exist. Metrics measure the failures you already knew about. A person reading twenty outputs finds next quarter's metrics.

• Recruiting Goodhart as an argument against measuring. Partly optimising the measure is manageable. Not knowing whether you are improving is not.

Interview Corner

Summary

• Evaluation is what makes AI engineering engineering. Without it you have software practices wrapped around a component nobody can characterise.
• Four obstacles, four responses: non-determinism (measure a distribution), no oracle (score properties, not identity), unbounded inputs (sample real traffic), a moving target (pin versions, re-evaluate on provider upgrades).
• Stop looking for ground truth on open-ended output. Required facts, citations, length bounds, forbidden claims — computable properties are a usable definition of good enough.
• Four jobs: attribution, regression detection, release gating, and — the one nobody expects — specification. Writing cases is where a team discovers it disagrees about the product.
• The ladder is L0 vibes → L1 golden set → L2 graded → L3 gated → L4 closed loop. Most teams are at L1 and describe themselves as L3. No rung is expensive given the one below.
• Measuring without gating is a dashboard. That step is organisational, and it is where most harnesses stall.
• Cost is never the real reason. A serious run is about eleven dollars, a small one sixty-five cents; one regression investigation buys 277 runs.
• The eval set is the only artefact that appreciates. Prompts, models and frameworks get replaced; a corpus grown from your own incidents does not.
• Read twenty raw outputs before designing anything, and have someone who did not build it review a batch — builders grade on effort, not on the answer.
• Goodhart is real and manageable: hold out a set you never tune against, report a vector rather than a headline, grow the corpus from incidents, and keep a periodic human read. It is not an argument for staying unmeasured.

In the next chapter, Building an Eval Set, we get concrete about the corpus itself: where cases come from, how many you need, how to write expectations for output that has no single right answer, how to keep the set balanced and versioned as it grows, and why the cases you least want to include are the ones worth the most.