Counting Honestly: Ranges, Zero Failures and Paired Changes

AI evaluation numbers in planning documents are often single percentages: "the new prompt scored 94.5%," "twenty for twenty on the pilot." Each sounds like a fact, and each hides what a release decision depends on: how sure we are.

The rule I want you to leave with fits on one line. Report every rate as a range, gate automation on the low end of that range, size your test set from the bar you need to clear, and compare two versions only on the cases where they disagree. To claim 99% with zero failures you need 381 clean cases, not 50 and not 100. To judge a change you read the cases it changed, and you read every one it broke.

Everything below can be checked with a spreadsheet and a calculator. I will show the arithmetic as I go.

Four of five is not 80 of 100

Start with two basketball players. One makes 4 foul shots out of 5. The other makes 80 out of 100. Both shot 80%. Nobody who has watched basketball believes those two records carry the same information, yet that is exactly how AI test results get reported. A score of 80% on 5 examples and a score of 80% on 100 examples land in the same cell of the same dashboard.

The difference is evidence, not performance. So instead of a single number, draw a band around each result: the range of true success rates that are still believable after what you saw. For 4 of 5, that band runs from 38% to 96%. That is practically useless. You have learned that the system is not hopeless and not perfect. For 80 of 100, the band runs from 71% to 87%. Same score, very different knowledge.

The same thing happens at the top of the scale, which is where automation decisions usually live. Suppose your model gets 49 of 50 test examples right. The honest sentence is "it scored 98% on 50 examples." The rate you would see on thousands of real items could be anywhere from about 89.5% to 99.6%. Fifty examples cannot pin it down any tighter, and no amount of confident formatting in a slide changes that.

A perfect run is the extreme case. A shooter who makes 5 of 5 ties one who makes 100 of 100 on the scoreboard, but the band for five shots stretches from a bit over half to perfect, while the band for a hundred hugs the top.

Ranges you can compute, and gating on the low end

The range I use for a yes-or-no success rate is the score interval, in its two-sided 95% form. It behaves sensibly near 100%, where the margin of error most of us met in school breaks. That textbook margin shrinks to nothing when every case passes, as if a perfect streak proved perfection. The score interval does not make that mistake.

The intuition is simple. It acts as if you had seen about two extra right answers and two extra wrong ones, which pulls the centre in from the edge, and then it wraps a margin around that centre that shrinks as the count grows. For 49 of 50 the centre moves from 98% to about 94.6%. That intuition explains why the centre moves; the exact recipe is below.

Here is the calculation for x successes in n tries, with z = 1.96 (so z squared is about 3.84):

centre = (x + z^2/2) / (n + z^2)
margin = (z / (n + z^2)) x sqrt( x(n - x)/n + z^2/4 )
range  = from (centre minus margin) to (centre plus margin)

49 of 50:
  centre = (49 + 1.92) / (50 + 3.84) = 50.92 / 53.84 = 0.946
  margin = (1.96 / 53.84) x sqrt(0.98 + 0.96) = 0.0364 x 1.393 = 0.0507
  range  = 0.895 to 0.996, so 89.5% to 99.6%

Put those three lines in a spreadsheet with x and n as inputs and you can reproduce every range in this essay. Check yourself against 4 of 5 (38% to 96%) and 80 of 100 (71% to 87%) before you trust the sheet.

Under its assumptions, the procedure covers the true rate in about 95% of repeated samples. It is not a promise about one particular case.

Now the part that changes decisions. When an action is about to run without a human checking it, the question is what I am promising if I ship today. So gate on the bound, not on the score. If an action needs 98% before it runs alone, the lower end of the range has to clear 98%. For a success rate you hold up the bottom of the band. For a failure rate you hold up the top.

The lesson's example makes this concrete. A team builds a system that reads supplier emails and flags delays. Before seeing any results, they write down their confidence level and their bar: a 90% success rate. They test 20 emails and every one is right. Twenty of twenty is well over 90%, so the instinct is to ship.

Hold up the bottom of the band instead. When every case passes, the lower bound simplifies to n divided by (n + 3.84). For 20 of 20 that is 20 / 23.84, about 84%. That is under the bar, so this is not a pass yet. As more all-correct emails come in, the bottom of the band climbs like a staircase, and somewhere in the mid-thirties it finally clears 90% (0.9 x 3.84 / 0.1 is 34.6, so 35 clean emails).

That one rule stops a team from shipping on a lucky small sample. I treat it like any release check: a fixed rule written before the result (see also the release gate is the product).

There are two ways to defeat a bound gate without noticing, and the supplier example has both.

The first is peeking. "Check after every batch and stop the moment it clears" gives a lucky streak many chances to cross the line. Fix the sample plan before you look, or deliberately use a method built for sequential monitoring.

The second is subtler. The range assumes representative, separate tries, like fresh shots. Pulling more emails from one large supplier is the easy way to grow the count, but those emails may share one template and one habit, so twenty of them might behave more like a handful. Count suppliers and threads, not just emails, and treat repeated runs of one case the same way. To see how much this matters, rebuild the test many times by redrawing whole suppliers and watch the score wobble. That is bootstrapping by group.

It changes the sentence you say in the room. Not "we tested 200 emails." Instead: "we tested this many separate suppliers, we set the bar first, and here is the cautious end of the range." Somebody can make a release decision on that sentence.

Zero-failure sizing: 73, 189, 381

The bound gate gives you a design tool. If you know the bar, you can work out how large a clean test set has to be before the bound can clear it.

Take the all-correct lower bound from the previous section, n / (n + z squared), set it equal to the bar p, and solve for n:

n >= p x z^2 / (1 - p),  with z^2 = 1.96^2 = 3.8416

95% bar:  0.95 x 3.8416 / 0.05 = 72.99   round up to 73
98% bar:  0.98 x 3.8416 / 0.02 = 188.2   round up to 189
99% bar:  0.99 x 3.8416 / 0.01 = 380.3   round up to 381

Those are the counts with zero mistakes. Every one of those examples has to come back right. With a couple of errors you need more. Round up, always: 188 clean cases leave the bound just under 98%, which is the whole point of the gate.

The same arithmetic runs from the error side. To show an error rate below 2%, you need 189 clean cases. Below 1%, 381. Below half a percent, 765 (0.995 x 3.8416 / 0.005 = 764.5, rounded up). A team that wants to tell a reviewer "this action fails less than one time in two hundred" is signing up for a gold set of 765 independent, representative, correct results before the claim holds.

For a quick check without a calculator there is the rule of three. Zero failures in n independent tries puts the failure rate below about 3 divided by n, at roughly 95% confidence. Zero errors in 30 means below about 10%. In 100, below about 3%. In 300, below about 1%. Zero seen is not zero possible. A pilot that comes back spotless on 100 checks still allows failures as high as about 3%, and if failures at that level would hurt the people the system serves, the pilot is not finished.

The rule of three carries the same conditions as the range: representative, independent tries and no repeated peeking. Like the score interval, it is for one yes-or-no rate, not a blended score.

The size of your gold set comes from the bar, and the bar comes from the action. A drafted reply a person will read can run at a lower bar than a legal notice filed without review. I size each automated action separately, per category, and write the count next to the bar before any data comes in.

A short story shows why. A team tests a new prompt on 20 examples and it gets all 20 right. The old prompt got 18. They ship and announce a big win, and a month later production is no better. Twenty of twenty has an honest range that starts around 84%. They mistook a lucky small sample for a fact. Nothing in their arithmetic was wrong; they just never asked what 20 cases could support.

There is a second effect hiding in stories like that one. An unusually good or bad week is partly luck, and the next week drifts back toward normal whatever you do. That is regression to the mean. Pick your worst week, make a change, celebrate the next week, and some of that improvement was coming anyway. The defence is to compare old and new on the same examples, which is the next section.

Paired change: 12 fixed, 3 broken, and read all three

A team changes the prompt on a model that sorts incoming email. On 200 test emails, the old version got 90% right and the new version gets 94.5%. Everyone in the room is smiling. Higher is higher, and it is the same emails.

Same emails is exactly the useful part, and the two final scores throw it away. Think about taste-testing two soups. Someone who likes both bowls tells you nothing about which is better. Neither does someone who dislikes both. The only tasters who carry information are the ones whose verdict changes between bowls.

Emails work the same way. Most were right under both prompts and a few were wrong under both. Set those aside and look only at the emails where correctness changed. In this example, the new prompt fixed 12 emails and broke 3.

You can rebuild the full picture. Old right is 90% of 200, which is 180. New right is 94.5% of 200, which is 189. With 3 broken, 177 emails were right under both (180 minus 3), and 177 plus 12 fixed gives the new 189. That leaves 8 wrong under both (200 minus 177, 12 and 3).

The 177 and the 8 drop out of the comparison. What is left is 15 disagreements, split 12 against 3. I like that already, because those are emails a manager can open and read. Twelve wins and three losses is a conversation. Four and a half points is a slogan.

Now ask whether 12 against 3 could be luck. If the change did nothing, each disagreement would be equally likely to go either way, like a fair coin. The exact version of the paired test on disagreements counts how often a split at least that lopsided happens by chance:

15 disagreements, coin-flip null
ways to get 3 or fewer "broke":  1 + 15 + 105 + 455 = 576
all outcomes:                    2^15 = 32,768
one tail:  576 / 32,768 = 0.0176
two-sided: 2 x 576 / 32,768 = 0.035

large-sample check: (12 - 3)^2 / (12 + 3) = 81 / 15 = 5.4, p about 0.02

So a split like this turns up by luck only about 2% to 4% of the time, depending on the version of the test. That is convincing. It is not the probability that the result is noise; it says that under the coin-flip assumption, with independent email pairs, this split would be unusual.

The lesson then runs a comparison that I find more instructive than the first. A second team tries a different prompt on the same 200 emails and lands on the identical final score. Theirs fixed 18 and broke 9. Both teams end up 9 emails ahead (12 minus 3, and 18 minus 9). The gut says the second team has the bigger win because 18 fixes beats 12. It is the weaker result. The second team needed many more flips to get to the same place, and a split of 18 against 9 turns up by luck roughly one time in eight. More motion, less evidence. The final scores could never have shown you that.

Significance is still not the decision. Before anything ships, open all three broken emails. Suppose one is a legal notice that is now filed as routine. That single break might outweigh a dozen tidier fixes. The lab deliverable makes this explicit: a paired-test note that lists the three broken cases individually, not just as a count.

Then put the gain in context. Estimate the size of the improvement with an uncertainty range suited to paired cases, and account for messages that share a customer or template. The new prompt might also take more effort per email or run slower. If only the optimistic end of the estimate justifies the change, you have a hopeful story, not release evidence.

There is one more trap, and it looks like ordinary iteration. The test did not pass, so you tweak the wording and run it again, and keep going until one version clears the bar. Iterating is fine. Grading yourself on the same pile is the problem. Try 20 prompt variants where none is truly better, keep the best, and there is about a 64% chance one of them looks better by luck alone. Each null variant has a 0.95 chance of not looking significant at the 5% level, so none of 20 do with chance 0.95^20 = 0.358, and at least one does with chance 1 minus 0.358, which is 0.642.

Every extra try, every new metric and every slice chosen after peeking gives luck another chance. That is the problem of forking paths. The fix is boring and it works. Write down what counts as a win before you look. Tune on development data, the pile you are allowed to learn from. Keep a sealed test pile nobody has looked at, run the final choice on it once, and report how many things you tried. If the sealed test disappoints, that is an answer too. A score on material the system was tuned against answers the wrong question, which is the same instinct behind agents need an independent exam.

Stratified estimates: 95.2% against a raw 89.5%

The last calculation is a common one to get wrong, even for careful people.

Suppose your traffic is 80% easy items and 20% hard items. To get enough hard cases to say anything about them, you audit 100 easy items and 100 hard items. The easy ones come back 99% right (99 of 100). The hard ones come back 80% right (80 of 100).

The raw audit average is (99 + 80) / 200 = 179 / 200 = 89.5%. That number over-counts the hard items. They are half the audit but only a fifth of real traffic. You sampled them that way on purpose, which was the right audit design and the wrong way to report the result.

Weight each stratum by its share of real traffic instead:

stratum   traffic share   audited   correct   rate   weighted
easy      0.80            100       99        99%    0.8 x 99 = 79.2
hard      0.20            100       80        80%    0.2 x 80 = 16.0
total                     200       179       89.5%  (raw)   95.2% (stratified)

The stratified estimate is 95.2%, not 89.5%. Neither number is a lie on its own. The raw average answers "how did the audit sample do." The stratified estimate answers "how will the system do on the traffic it will actually see." Only the second one belongs in a release decision.

The error runs both ways. Oversample the easy cases and the raw average flatters you. Oversample the hard cases, as here, and it undersells you, which can kill a sound automation for no reason. So the rule is short: always weight by traffic, never just average the audit.

Keep each stratum's own range visible next to the blended estimate, because the hard stratum is where the bound gate will bite, and re-weight when the traffic mix shifts. The blend you report has to match the mix you serve, a theme that also runs through we measured local safety models, the hard part was deciding what works.

The sizing table to keep on the wall

The module's lab turns all of this into two deliverables, and both are easy to rebuild.

The first is a sizing table for every action you plan to automate. In a spreadsheet, make one row per action, with columns for the action, the category, the bar, and the zero-failure count computed as p x 3.8416 / (1 - p), rounded up. Add a column for the clean count you actually have and a column for the lower bound it gives you. Your sheet should reproduce these values:

If your sheet says 188 or 379 anywhere, it came from rounding down or an older approximation, and the bound at that count sits just under the bar (188 / 191.84 is 97.998%).

The second deliverable is the paired-test note. Make a sheet with one row per test case: case id, category, old correct (1 or 0), new correct (1 or 0). Add a "fixed" column that is 1 when old is 0 and new is 1, and a "broke" column that is 1 when old is 1 and new is 0. Sum them. With the provided result set you should get 12 fixed and 3 broken, out of 200 cases (177 right under both and 8 wrong under both), and an exact two-sided p of about 0.035 (2 x 576 / 32,768). Then filter on the broke column and write down the three cases, what each one was, and how much harm each break would do in production. That list is the part a reviewer will actually read.

To check the stratified estimate, add a third small sheet with traffic share, audited count and correct count per stratum, and confirm you get 95.2% weighted against 89.5% raw.

If I compress the module into what I would print and pin up, it is this. A score without a range is a guess. A gate on the point estimate is a gamble. Report a range, gate on its lower bound, size the test set from the bar, compare versions on the same inputs, and read every case the new version broke.

Go deeper

The two lesson videos behind this essay are in the engineering video library: "A perfect small test is not proof" (v06) walks through clean runs, the band around a score, the bound gate on the supplier emails and the rule of three, and "Did the change really help?" (v17) walks through the 12 fixed against 3 broken comparison, the second team's 18 against 9, and forking paths. The module runs 9 minutes of video plus a 35 minute lab.

Go deeper: the full lesson and lab are in the Transition2 course.