AI evaluation and measurement / Field essay 81

Working Back From a KPI to a Gold Set and Per-Step Evals.

How I turn one business number into an outcome label, a severe-error label, a blind gold set and a check at every step, and why the chain math decides the plan.

Published

ESSAY
FIELD ESSAY 81 / AI EVALUATION AND MEASUREMENT

Working Back From a KPI to a Gold Set and Per-Step Evals

"We know which business number we want to move. How do we turn that into something we can test an agent against?" Engineering leaders keep asking some version of this, usually about a flow with several steps or several agents. My approach is to work backwards from the number until I reach something a test can check, and not to start building until I get there.

This essay covers the reasoning behind that walk and the traps along the way. The step-by-step version is on Transition 2, and related essays are collected under evals and measurement.

Start from the number, not the stack

Most projects start from a model, a framework or a demo. All three put off the question "what does good look like?", and the answer never arrives later in a form a test can check. Progress reviews then rest on how the demo felt.

The first thing I ask for is one problem and one number. If three KPIs are in play, I ask which one the work exists to move, and I set the others aside. Then I ask for two sentences: the outcome that moves the number, and the outcome that must never happen. The first becomes the success label and the second becomes the severe-error label. Every eval I build later traces back to those two sentences.

The trap here is choosing a signal that is easy to collect instead of the outcome itself. Approval is the classic case. A workflow can run at 99% approval while a blind audit of 200 items finds 17 wrong. I wrote about that gap in a 99 percent approval rate is not an accuracy number. If the label is "a reviewer accepted it", the eval measures the reviewers.

Why the answer key comes first

Once the outcome is written down, I want the gold set built before the system exists, or at least before anyone has seen its outputs. The reason is anchoring. A person labelling with a model's answer in view drifts toward it, and the key quietly becomes a record of what the model said. I make that argument at length in your answer key is the product.

The construction rules I hold to are:

  • real cases
  • two labellers working blind
  • a written guideline with a version number
  • stratification by slice
  • a time split, so test items come from later than dev items

Teams usually skip the time split, yet it is what makes the test look like the future rather than the past.

The objection about effort comes up early. 400 items with two labellers at one minute each is 800 minutes, about 13 hours. In my judgement, that is small next to the time a project spends arguing about whether it works when it has no answer key.

Disagreement is the guideline talking

Two labellers will disagree. The tempting move is to have a third person break ties and move on. I don't, because the disagreement tells you something about the guideline.

I compute Cohen's kappa first. Take 100 items where both labellers pass 80, both fail 12, and they split 5 and 3. Raw agreement is 0.92, chance agreement is about 0.731, and kappa is about 0.70. I read that against rough bands at 0.6, 0.7 and 0.8, but I care more about the split cases than the number. Each split case gets a written boundary rule, and guideline v1 becomes v2. A low kappa means the definition of "good" was not yet precise enough to build against. Finding that out before the build is the easiest time to fix it.

The chain math decides the plan

For a multi-step flow, the most useful number in the whole exercise is the compounding one. Five steps at 95% each complete about 77 of 100 runs. Ten steps at 95% complete about 60. Twenty steps at 98% complete about 67.

That math changes what I build, not just what I measure. It tells me to remove steps where I can. It also tells me to turn any step that plain code can do with certainty into plain code, because a deterministic step doesn't add error to the chain. And it tells me the eval has to label intermediate results: what each step should produce, not only what the chain should end with. Without those labels, an end-to-end failure tells you something broke but not where.

Each step then gets the metric that matches its mistakes. A flagging step is judged by what a flag means at the real base rate. A flagger whose flags are real only about 1 time in 6 (190 of 1,188) should send items to review, not block them. Counts out of a thousand walks through that arithmetic.

Size from the bar, and route the thin slices

Once the bar is written as a lower bound, the test size follows from it. With zero failures, claiming 95%, 98% or 99% takes 73, 189 or 381 clean items. I apply this per slice. I treat a thin slice as a routing decision rather than a gap to cover over: a slice with fewer than 73 clean items stays human-reviewed until it has the evidence. That keeps the strong slices moving without pretending the weak ones are proven.

The judgement call here is resisting the urge to merge thin slices into a bigger one just to reach the count. Merging is only honest when the slices really behave alike. Otherwise it hides the slice most likely to fail.

What "on track" should mean

Mid-project, I judge progress on two things. One is the gold-set pass rate per slice, with its Wilson range. The other is the list of steps that still need a person. Lower bounds rising on a fixed held-out set, and that list getting shorter, means on track. A better demo with an unchanged table does not. Defining this in advance matters as much for agents reporting status as for people. Landed is not done applies the same discipline to agent work.

Questions people ask

Where should an AI eval start?

With one business number and two written outcomes: the one that moves the number and the one that must never happen. Those become the success label and the severe-error label that every later check traces back to.

Why build the gold set before the system?

Because labels drift toward whatever answer the labeller has already seen. A key written before any model output stays independent of the system it grades, which is what makes it a fair test.

How do I report progress on an AI project mid-build?

Report the gold-set pass rate for each slice with its Wilson range, plus the list of steps that still need a person. Rising lower bounds and a shrinking list mean progress; a better demo alone does not.

If you are working through this on your own flow, connect with me on LinkedIn to compare notes and discuss.

CONNECTED READINGForward and back
Keep reading

The answer deserves
another experiment.

Back to the journal