A lucky week is not a better model.

A lucky week is not a better model

A pilot has a great week. The new prompt scores 98% on routing work, against a usual level of around 90. Leaders announce a rollout. The next week, same prompt, same system, it scores 91.

Nothing broke. That first week was chosen because it stood out, and part of what made it stand out was luck: which messages happened to arrive, which cases happened to get measured. When you measure again, the luck does not come along.

This essay covers four ways luck dresses up as progress in AI evaluation, and the one habit that handles all of them. The four are the week after an extreme week, the winner of many tries, the experiment stopped on a good day, and the comparison run on different cases. The habit is to decide how you will measure before you look at any result.

The numbers below are teaching numbers from four course lessons: regression to the mean, the repeated-test trap, sequential stopping, and the paired comparison lesson.

Every score is true level plus luck

Start with the simplest picture. Any measured score combines two parts: the system's true operating level, and sampling variation from which cases got measured. Take enough basketball shots and one lucky bounce drops in. A score works the same way.

The lesson's routing team ran a new prompt for several weeks and highlighted its single best week, 98% accuracy. Its usual score sits around 90. The following week came in at 91.

The tempting story is that something broke between the two weeks. The likely story is that 98 was a high draw and 91 is what an ordinary draw looks like. An extreme result, picked because it was extreme, usually lands closer to the long run average when you measure again. That is regression to the mean.

Two things it is not. It is not a force acting on the model. And it does not guarantee that a bad week is followed by a better one; it only makes an ordinary week the likely next result. The logic runs in both directions, and the downward direction fools people more often than the upward one.

The lesson's closing exercise makes the point. The three worst-performing support agents, the people answering tickets, got extra training and improved the next month. Did the training work? You cannot tell yet. They were picked because their scores were unusually low, so they would likely have improved somewhat anyway. The fair check compares them with similar agents who got no training over the same period.

The same trap shows up in release reviews. A team picked for a terrible week improves on its own, and the coaching takes the credit. A high pilot softens, and a new release takes the blame. To tell change from noise, I want three things: a concurrent comparison, repeated measurements, and an analysis written down before the data arrived. I also record workload, case mix and label changes, because a score can move simply because the population being scored changed.

Twenty prompts, and the winner is partly lucky

Now take selection one step further. Picture twenty prompt versions that are truly about equal, each tested on a small batch of examples. Pure chance spreads their scores out. Some land high, some land low, and none of the spread is real. Promote the top scorer and you have picked a prompt that is partly just lucky.

In the lesson, a team crowns that winner and claims a gain of about a dozen points over a baseline. Two weaknesses sit in that one sentence. The winner was chosen for luck. And the baseline was scored on different examples, so the comparison is not fair either. The repair is to rerun both versions on the same fresh examples.

The repeated-test lesson makes the same point at a larger scale. A team has a 200-question validation set and tries 60 prompt variants. After each result they read the misses and revise the prompt around them. Eventually one version scores 98%.

By then the validation set has quietly become part of the development process. The wording, the thresholds and the examples were all chosen to fit those 200 questions. The 98 is partly a record of search effort, not a clean estimate of how the prompt handles new cases. No model was trained on the set. Human selection adapted to it, and that is enough.

Every extra try is another lottery ticket against the same answer key. Enter enough of them and one will look strong partly by luck. That is the winner's curse, and it grows with everything you try: prompts, seeds, model versions, thresholds, and every decision made after looking at results. A confidence interval computed as if the final prompt had been chosen in advance is too optimistic, because it ignores the search.

The lesson's quick check: a colleague ran a model twelve times with different random settings and reported only the best score. Trustworthy? It is the luckiest of twelve tries, so it overstates typical performance. Report all twelve, or freeze one setup and confirm it on fresh cases.

Cutoffs fall into the same pattern. A team tries 40 cutoffs on the same labeled month, picks the one with the fewest misses, then reports precision and recall from that month as though the cutoff had been fixed in advance. On a fresh month the chosen point usually looks less favorable, because it no longer has the luck of being the best of 40. That month is now development data. Freeze one operating point and measure it on later months.

What I keep for any search:

• a development set where the team can try anything it likes,
• a holdout kept sealed from the people writing prompts until the choice is frozen,
• a search log: what changed, how many versions were tried, which cases influenced the changes, and which set is still sealed.

Leaks can be innocent. Someone pastes a missed holdout example into a few-shot library, or rereads the exact error list. The label file stays technically separate, but the team has learned the answers. Once a holdout has been opened and used to revise, it is development data. Label a fresh set before making a new claim.

Peeking every day and stopping on a good one

The third trap is time. A product team checks a live experiment after each day. On day three the new flow is ahead, and the team stops early to ship. If the difference had gone the other way, they would have waited another week.

That flexible finish line is like ending a race whenever your runner happens to lead. Each look is another chance for a random swing to produce a favorable headline. A standard test is calibrated for one planned look at a defined sample size. Reuse its pass line every day, stop the first time it clears, and the chance of at least one false win climbs with every look.

This is not an argument for ignoring new information. It is an argument for a decision rule built for repeated looks, rather than one borrowed from a one-time test. Such methods exist: a sequential procedure that divides its error allowance across the planned looks, or a confidence sequence that stays valid however often you check. Whichever you choose, name what the boundary means before you use it.

Optional stopping hides in softer forms too. Running until "the numbers look stable." Switching the main metric after a weak result. Stopping once one subgroup crosses a line. Each of these is an extra test. I log every planned look, metric and slice, and keep a record of deviations. If the plan changes for a genuine operating reason, the analysis gets labeled exploratory and confirmed on fresh evidence.

There is one important exception, and it is about safety, not benefit. An operator can pause a system after a credible security incident even when the statistics are incomplete. That is a precaution, not a claim. Peek as often as you like for danger. What you cannot do is stop the moment the scoreboard flatters you and call it a win.

The lesson's worked plan keeps the two apart. A pilot enrolls 2,000 cases over four weeks. The owner wants weekly safety checks and one final quality comparison. Before the first case, the plan writes down the weekly monitoring method, the hard harm stop, the final quality question and the sample size. A hard safety event triggers containment right away; it does not wait for a significance threshold. The final comparison follows its own planned rule.

A second version runs longer: a six-week rollout with a weekly random sample of 100 actions and a final comparison after 600. One critical wrong action can trigger a pause. Trends in ordinary errors follow a preset boundary. The pause log and the benefit analysis are kept separately, so an operational response is never reported as a statistical discovery. The plan also says how cases with late labels count while the team waits, and it carries a maximum duration so the experiment cannot run forever.

And the lesson's scenario: a test looks great on day four, and someone wants to announce it. Stick to the planned stopping rule. Unless the plan allowed a day-four look with an adjusted bar, keep collecting until the declared finish.

Same cases, and count only the disagreements

The fourth trap is comparing on different cases. A team that tests each version on its own random sample mixes the change it made with how hard the cases happened to be. A large headline difference can be noise.

The repair is to run old and new on the same cases, and the paired comparison lesson shows what pairing gives you. Two extraction prompts read the same 200 supplier documents and score almost the same overall. For every document, record whether A was right and whether B was right. That sorts the documents into four groups:

Shared wins and shared misses together come to 175. Those cases describe the common baseline, and they still matter for the overall service bar. But they say nothing about which version is better, because both versions did the same thing on them. Only the 25 disagreements carry the comparison: 18 where only A was right, and 7 where only B was right. A document where the two answers differ but both are wrong does not count as that kind of disagreement.

Then read the 25. Perhaps B fixes date formats but breaks handwritten fields. Sort the disagreements by source, field and severity, and the mechanism shows up. A sensible decision might keep B for the date lane while handwritten fields stay with A or go to a person. A test can detect a consistent difference without showing that the difference is worth deploying. You still have to weigh the harm on each side.

Pairing only works when the comparison is controlled. Both versions see the same eligible cases, under the same label definition, with identical preprocessing. If one prompt gets a cleaner scan, the comparison no longer isolates the prompt. Keep case IDs stable, and before blaming a model for a regression, check the label and the source, because an ambiguous rubric can create a flip too.

This ties back to the first section. The regression lesson's version of a real gain: a new prompt reaches 98% on a sealed holdout, and the old prompt reaches 91% on those same cases. That seven point paired difference is much stronger evidence than a 98 picked from many tries. Real improvements survive a retest on the same sealed cases. I still break the seven points down by intent and look for regressions on ordinary routing.

One more limit. The paired test answers whether two versions differ on these records. It does not say how next quarter's traffic will behave. Use the comparison to locate regressions, and keep a separate, later set for launch evidence.

The fix: write the rules down before you look

All four traps share a root. A number was chosen after the data was seen: the best week, the best prompt, the best day to stop, the most flattering comparison. The repair is the same for all four. Decide the measurement first.

Before the first result, I write down:

1. The sample size, and where the cases come from.
2. The stopping rule: when we look, what each look is allowed to decide, and the maximum duration.
3. The bar for a win and the method behind it, including a method built for repeated looks if we will look more than once.
4. The comparison: old and new on the same cases, with the disagreement counts reported, not just two percentages.
5. The safety stop, kept separate from the benefit claim.
6. The search log, and which set stays sealed until the choice is frozen.

Then, when a score looks unusually good or unusually bad, three questions come before any celebration. Why was this sample measured? How many versions were tried? Was the next check independent?

If the result is marginal, collect another planned batch rather than announcing a win or picking the most favorable week. A limited, reversible pilot can run while the evidence matures. The marketing claim waits for a design that supports it.

None of this says improvements are fake. Regression to the mean is a warning about selection and noise, not a claim that every gain is luck. A real improvement repeats under a fair test. A lucky week does not. The whole job is building a test that can tell them apart.

Go deeper

The four lessons behind this essay, regression to the mean, the repeated-test trap, sequential stopping and the paired comparison lesson, each work through these examples step by step.

Go deeper: the full lessons are in the Transition2 course.