AI evaluation and measurement / Field essay 80

Why Passing Evals Still Fail in Production.

A passing eval is usually a point estimate on tuned items with one blended score. Here is how I take it apart: lower bounds, severe-error lines, slices and judges.

Published

ESSAY
FIELD ESSAY 80 / AI EVALUATION AND MEASUREMENT

Why Passing Evals Still Fail in Production

"Our evals pass. Why does production still surprise us?" Engineering leaders keep asking this. They usually sense the number flatters the system but can't say how. My short answer is that most evals are a point estimate wearing a suit. The suit is the dashboard, the trend line and the decimal place. Underneath is one rate, measured once, on items that were also used to tune the thing being measured.

This essay is how I take that apart, and the judgement calls I make along the way. The practical checklist is on Transition 2. Related essays are collected under evals and measurement.

Taking 49 of 50 apart

When someone shows me 98%, the first thing I ask for is the count behind it. Say it is 49 of 50. The Wilson score interval for 49 of 50 runs from about 89.5% to 99.6%, and its centre is near 94.6%, not 98%. With small samples the interval pulls toward the middle, because a single extra failure would move the rate a lot.

So the evidence supports "at least about 89.5%", and that is the number I gate on. The same logic makes 4 of 5 and 80 of 100 very different claims, even though both read as 80%. The first spans 38% to 96%. The second spans 71% to 87%. I covered the general idea in a rate is a belief with a width.

The judgement call is when to set the bar. I write it as a lower bound before anyone looks at results. A bar set afterwards tends to land just under whatever came back.

Was the test set also the tuning set?

The second thing I ask is whether any of the 50 items were used to choose a prompt, a threshold or a retrieval setting. Usually the honest answer is yes, all of them.

That matters more than people expect. Try 20 variants that are truly equal, and there is about a 64% chance at least one looks better on your set by luck. If you pick the winner on the same items you report, the report contains the luck. Re-run that winner on fresh items and it usually falls back, which teams then misread as the model getting worse. A lucky week is not a better model covers the same trap over a different time span.

My rule is blunt: a held-out set is for reporting, never for choosing. The subtle failure is not deliberate peeking. It is the engineer who reads three held-out failures to understand them and then edits the prompt. Those three items are now dev items. I retire them from the held-out set and replace them, rather than pretend the set is still clean.

One score trades things that should never be traded

A blended score counts a formatting slip and an action on the wrong record as one failure each. A change that fixes ten slips and introduces one wrong-record action therefore shows up as an improvement. That is the most common way a passing eval leads a team into a production surprise.

So I ask the business owner for the absolute no's and the rate they will accept for each, and I give those their own gate. The severe gate is checked first, and gains in the overall score never offset it. Which errors count as severe depends on what a mistake does next, so I treat it as a business decision rather than a modelling one. See pick the metric by what a mistake does next.

The arithmetic then says how much evidence the gate needs. The rule of three) gives a quick read: zero errors in 300 items puts the rate below about 1%. When I size a real gate I use the stricter Wilson-based table, which needs zero severe errors in 189, 381 or 765 items to show a rate below 2%, 1% or 0.5%. A set of 50 with zero severe errors has shown very little, and I say so in those words.

Averages hide the slice that fails

Even an honest overall number can hide a failing slice. If one slice passes 99% and carries 80% of traffic, and another passes 80% and carries 20%, the traffic-weighted rate is 95.2%. That figure is correct, and it still hides that one request in five fails in the second slice. Accuracy is worse for rare events: a detector can show 99.9% accuracy while fewer than 9% of its flags are real.

So I report precision and recall per slice and per action. My rule for where to cut slices is simple: split wherever the inputs differ in kind or a mistake does something different. If I can't say why a slice exists, it probably doesn't need its own line.

Consistency is its own property

One run per item measures how often a step is right, not whether it is reliably right. A step that passes 90% of the time passes 8 runs out of 8 only about 43% of the time. Even at 99% per run, 8 of 8 is about 92%.

For read-only steps, I don't worry much about this. For anything that acts, I want pass^k reported next to the single-run rate, because production repeats the step far more than 8 times.

The judge is an instrument

Most eval setups now have a model grading the outputs. That model is a measuring instrument, and it needs calibrating like any other.

The trap is raw agreement. Suppose that on 1,000 outputs the judge passes 970, people pass 960, and they agree on 950. That looks like 95% agreement. But given how often each side says pass, chance agreement is already about 0.932, so Cohen's kappa is (0.95 - 0.932) / (1 - 0.932), about 0.26. Most of the agreement is both sides passing easy items. On the failures, which are the point of the exercise, the judge is weak.

Leniency makes this worse. Say the judge flags 30 failures in 1,000. If it catches only half of the real failures, there are about 60 bad outputs, or 6%. If it catches 30%, there are about 100, or 10%. The judge's pass rate tells you nothing until you know its catch rate.

So I check every judge against blind human labels, both overall and per slice. I swap answer order and count the flips; 28 flips in 100 means position is deciding the verdict. I seed outputs that must fail and void any run where one passes. The principle behind all of this is the one in why agents cannot grade their own work: the grader has to be independent of what it grades, and you have to measure it.

Comparing two versions

When a new version arrives, I compare it with the old one only on the items where the two disagree. If it fixes 12 and breaks 3, an exact paired test gives a two-sided p of about 0.035. That is decent evidence of improvement.

Then I read the 3 broken cases. That is a judgement call I won't hand to a statistic. If one break crosses the severe line, the answer is no, because the two gates don't trade against each other. Counting honestly goes through the paired test in more detail.

What I hand a decision-maker

None of this produces a single number. For each action, I give a decision-maker:

  • the bar
  • the lower bound for each slice
  • the severe gate status and its sample size
  • pass^k for steps that act
  • the judge's kappa and control results
  • the paired result, with the broken cases listed

It takes longer to read than "98%", but it is the version that predicts production.

Questions people ask

Why is 49 out of 50 not 98% accuracy?

98% is the point estimate. The Wilson range for 49 of 50 runs from about 89.5% to 99.6%, so the evidence supports a rate of at least about 89.5%. That lower bound is what a release gate should use.

What is pass^k?

It is the share of items that pass on every one of k repeated runs. A step that passes 90% of the time passes all 8 of 8 runs only about 43% of the time, so single-run scores overstate reliability for steps that act.

Why can an LLM judge agree with humans 95% of the time and still be unreliable?

When almost everything passes, chance agreement is already high. A judge that passes 970 of 1,000 against humans passing 960, agreeing on 950, has a kappa of about 0.26, which is weak on the failures that matter.

If you are working through the same problem, connect with me on LinkedIn to compare notes and discuss.

CONNECTED READINGForward and back
Keep reading

The answer deserves
another experiment.

Back to the journal