An eval that runs on every change
A release gate that has never failed tells you very little. It might be guarding the release. It might also be a green light wired to nothing, and from the outside the two look identical. Every build passes, the board stays green, and nobody can say whether the checks would have noticed a real defect.
The fix is not more checks. It is a gate that has to prove two things on every change: that the exact thing you ship is the thing that passed, and that the checks can catch a mistake you planted on purpose. If either proof is missing, the green result does not count.
This essay draws on three lessons from the Transition2 course: release contracts and negative controls, seeded failures and order swaps, and the four-part verification bar. The examples and figures are the lessons' own.
What the gate has to prove
The release contract lesson puts the rule in one sentence, and it is worth keeping on the wall:
A release gate must prove the exact deployed artifact passed required checks and that the checks can detect a known defect.
Two halves, and a gate can hold one without the other. The first half is about identity. A dashboard can show every test passing while a later build step packages different files and deploys those. Same source commit, possibly different contents. Rebuilds can change dependencies, generated files or packaging while the commit label stays the same. A green report on yesterday's box does not certify the different box shipped today.
The second half is about sensitivity. A suite that never fails when a meaningful defect is planted may not enforce anything. The lesson's phrase for that is blunt: a gate that has never demonstrated detection may be decorative rather than protective.
An eval that runs on every change is the habit of checking both halves every time, not once when the pipeline was built.
Four buckets for every graded answer
Many evals grade answers: an automated checker, sometimes an AI model acting as a judge, decides whether each output is acceptable. The seeded failures lesson shows how to tell whether that checker is measuring anything.
Mix two kinds of known material into the run. Seeded failures are answers broken on purpose, with a known expected verdict of fail: a wrong citation, a missing required fact, an unsupported number, a prohibited action. Valid controls are answers known to be good. Then every graded control lands in one of four buckets.
| The control was | Checker said fail | Checker said pass |
|---|---|---|
| A seeded failure | Caught, as it should be | False acceptance: the defect got through |
| A valid answer | False rejection: good work blocked | Accepted, as it should be |
Each bucket protects against a different way of fooling yourself.
The caught bucket is the one people report. On its own it proves nothing, because a checker that rejects everything catches every seeded failure. The lesson makes exactly this point: a judge that catches every planted mistake still has to show it accepts the valid answers, or it simply buries reviewers.
The false acceptance bucket is the one that matters when a defect reaches users. Every entry there is a known bad answer the gate let through.
The false rejection bucket is the one that wears a team down. A checker that rejects many good answers overloads reviewers and slows the release.
The accepted bucket keeps the checker honest from the other side. Reject-all behavior fails the good controls. Easy-only success fails to test sensitivity. You need both kinds of control in every run, or one of the failure modes is invisible.
So the gate reports two numbers side by side, false acceptance and false rejection, and it reports them per defect type, not as one blended pass rate.
Read the denominator, per defect type
A single overall number hides the gap that matters. The seeded failures lesson gives the example: 40 seeded failures, 15 of them wrong-source citations, and the judge catches only 4 of those 15.
Do the arithmetic the overall rate invites. Suppose, for the sake of the working, the judge caught every one of the other 25 seeded failures. The total would read 29 of 40. That sounds like a judge that mostly works. Split by type, it reads 25 of 25 on the easy defects and 4 of 15 on wrong-source citations. The second line is the real finding: on the defect this check exists to stop, it misses 11 of 15. The overall rate is not reassuring, and the targeted miss exposes a broken control.
Three habits follow.
Report the denominator for every seeded control. "Caught 4" means nothing without "of 15". A small count gives a wide range, and the reader needs the count to judge the width.
Keep the defect types separate. A judge that catches citation errors but accepts an omitted exception needs a narrower scope or a better rubric, and only a per-type table shows that.
Keep uncertain verdicts out of the automatic gate. If the checker cannot settle a case, a reviewer resolves it before it counts as a pass or a fail. Where serious examples are scarce, the checker's score stays advisory.
The seeded negative control
The negative control is the part of the gate that tests the gate. In a disposable copy of the release, introduce a known defect, run the required checks, and confirm they fail. The release contract lesson's example is a planted permission bypass: the unit, integration and authorization checks run against the release build, and the planted bypass must make the control fail.
Read the result carefully, because the wording runs backward. The control failing is the good outcome. It shows the suite can detect the boundary it claims to protect.
The control passing is the alarm. If a planted defect sails through, the check may be decoration. The lesson gives the response in order:
- Block the release.
- Repair the test or the enforcement point.
- Repeat the control and confirm it now fails, and that it fails for the right reason.
- Rerun the whole suite on the pinned release build.
The third step is the subtle one. A control can start failing for an unrelated reason, a typo in the fixture or a crash in setup, and look like a repair. Confirming the reason is what separates a fixed check from a check that broke in a new way.
Seeded controls also wear out. If every planted mistake uses the same giveaway phrase, the judge can learn the phrase instead of the rule. If the same seeds are visible during prompt tuning, the prompt can learn the seeds. The seeded failures lesson's countermeasures:
- Vary the wording and the position of each planted defect.
- Keep a sealed control set that never appears during tuning, alongside a rotating hidden set.
- Shuffle the answer order and hide model names, since neither should change a verdict.
- After any change to the judge, inspect the remaining misses and check that supported answers are still accepted.
A seeded defect is a fire drill. The real fire may look different. That is a reason to vary the drills, not to skip them.
Bind the result to the exact build
The identity half of the contract is mechanical, and that is its strength. Build the release once. Compute a digest, a fingerprint of those exact files. Run the required checks against that build. Deploy the same digest, and store the evidence linking the test result, the digest and the release.
The release contract lesson walks one case. A service build receives digest D. Unit, integration and authorization checks run against D, and the planted permission bypass makes the control fail, as it should. The release system deploys D and records the digest. A reviewer can then verify the chain without relying on a copied green badge.
Then a small packaging edit produces digest E after D passed. Same source commit, new contents. The rule is plain: E is a different artifact and must pass the required gate itself. Do not infer that E is covered because the source and commit labels match. Rerun the checks against E, or return to a reproducible build that yields D again. If neither is possible, hold the deployment, or roll back to a known-good build whose evidence is intact.
The deployment step enforces this, not a person's memory. It accepts only a digest that has passing evidence attached. Any change after testing creates a new identity, and the gate runs again.
Four questions one green suite cannot answer
A gate that runs on every change is one witness. The verification bar lesson is careful about what that witness can and cannot say.
It checks a change four ways: against the request, against the changed code, with a meaningful test, and through the running behavior. These are different questions. A test suite can pass while the screen still loses a user's draft on a timeout. A passing suite also does not show that a cross-account request is refused, which is why a separate reviewer tries one. The same passing unit test cannot stand in for all four.
Two parts of that lesson belong inside the automatic gate.
A failing control first. Before the fix, write a test that fails on the old behavior and should pass after the change. Then, once it passes, check that nearby bad input is still rejected, so a too-permissive shortcut cannot fake success. A test that was never seen failing has not shown it can.
No self-approval. If the same agent writes the code, the tests and the approval story, all three share one blind spot. Separate reviewers test separate claims: one inspects the exact changes, one exercises the live flow, one probes for abuse and cross-account access. Each finding stays open until an accountable owner resolves it or knowingly defers it, and it closes only with a linked retest.
The automatic gate does not replace those reviews. It makes sure every change arrives at them with the basic proofs already attached.
Wire it so red stops the merge
A gate that reports but does not block is a dashboard. Wiring it is mostly a matter of writing the contract down before the release, not during the argument about it.
Name the required checks. Name the severity thresholds, the evidence to keep, the emergency approver and the rollback procedure, before anyone sees the results. Then connect the parts:
- The merge requires the gate to pass. A red result stops the merge, and there is no "merge anyway" path outside the named emergency approver.
- The gate reads its checks from the release build by digest, not from a commit label.
- The negative control runs as part of the gate. If the control passes when it should fail, the gate is red.
- If a required check cannot run, the release holds. A check that did not run is not a check that passed.
- Every bypass is recorded with its reason, and the team reviews whether planted defects keep getting caught.
The release contract lesson's point about tracking is practical. When a test is flaky or unavailable, the release owner has a clear response instead of a temptation to treat every green status as equivalent. Over time the bypass log and the control history show whether the gate still protects anything.
One more wire. A rubric update can silently change what the score means. Version the instructions, the label definitions and the challenge set together, and run the old and new judges on the same blinded examples before the new one becomes the gate. Settle what counts as an error first, then measure the judge, and only then let it change a release decision. A judge approved under one rubric does not inherit approval under a revised one.
A short checklist
- Build once, fingerprint it. Run every required check against that digest and deploy only that digest.
- Any change after testing is a new build. Rerun the gate, return to the verified digest, or hold and roll back.
- Mix seeded failures and valid controls into every run, blind, with varied wording and position.
- Count all four buckets. Report false acceptance and false rejection side by side, per defect type, with the denominator.
- Plant a defect in a disposable copy and require the gate to fail. If it passes, block, repair, confirm it fails for the right reason, and rerun.
- Keep a sealed control set that tuning never sees.
- Keep uncertain verdicts out of the automatic gate until a reviewer resolves them.
- Write the contract first: required checks, thresholds, evidence, emergency approver, rollback.
- Make red block the merge, treat a check that did not run as a hold, and record every bypass.
- Version the rubric, labels and challenge set together, and compare old and new judges on the same examples before switching.
For the wider argument that the gate is where release quality actually lives, see the release gate is the product, and for the narrower question of proving the tests ran at all, a merge gate must prove the tests ran.
Go deeper: the full lessons and labs are in the Transition2 course.