Agent fleet field notes / Field essay C1

Ratchet the gate; do not hand-bump the floor.

A release gate with a typed-in minimum test count goes stale in one of two directions. Derive the floor from what exists, let the best result only move up, and make lowering it a recorded decision.

Published

RELEASE ENGINEERING AND TEST EVIDENCE
FIELD ESSAY C1 / AGENT FLEET FIELD NOTES

Ratchet the gate; do not hand-bump the floor

How do you stop a release gate that counts tests from rotting? Do not type the minimum into the gate. Derive it from the tests that exist, remember the best result you have ever seen, and refuse anything below that memory unless a person lowers it on purpose. That last part is a ratchet: it turns freely upward and does not slip back.

I came to this from a nightly run where a gate's floor was a number someone had computed once, as three quarters of the declared test ids, and typed into a config. Every pull request that added tests made the number a little more wrong. It broke three times in one night, and each time the fix was a separate change to a constant, reviewed by hand. The loud failure was annoying. The quiet one worried me more.

The two ways a typed-in floor fails

A literal floor goes stale in two directions. If the suite grows and the floor does not, the gate stops noticing losses: a run that silently skips ten tests still clears a bar set when the suite was smaller. If the floor is set from a ratio of ids and the id list changes shape, the gate blocks good runs and someone bumps the constant to get unstuck. Bumping in a hurry is how a floor ends up lower than it should be.

I wrote a small lab to see the first failure with numbers rather than argue it. The suite grows from 120 to 150 declared ids over five nights. The literal floor stays at 90, which was 75% of 120. On the fifth night a broken runner executes 120 tests instead of 130. Here is what each gate says:

run  ids  fixed-floor      ratchet
100  120  pass (>= 90 )   pass (>= 90)
110  130  pass (>= 90 )   pass (>= 100)
120  140  pass (>= 90 )   pass (>= 110)
130  150  pass (>= 90 )   pass (>= 120)
120  150  pass (>= 90 )   FAIL (>= 130)
130  150  pass (>= 90 )   pass (>= 130)

The literal floor passes a run that lost ten tests. The ratchet fails it, because the best earlier run executed 130. Note that 120 also clears the ratio rule on its own (75% of 150 is 112), so only the remembered best can catch this loss.

The rule in four lines

derived = int(0.75 * ids)
floor = max(derived, state.get("best", 0) - state.get("slack", 0))
ok = ran >= floor
if ok: state["best"] = max(state.get("best", 0), ran)

The floor is the larger of the ratio rule and the best passing run, minus an explicit slack you choose, which is zero in the lab. The best value only changes on a passing run, and only upward. A failing run cannot lower the bar, and a flaky night cannot raise it, because only passes count.

What a ratchet costs you

A ratchet will block a legitimate drop. Delete a test file on purpose and the next run fails. That is the correct behavior, and it needs a legitimate exit: a command that lowers the recorded best, takes a reason, and writes the old and new values somewhere a reviewer can read. Without that exit people will edit the state file, and you have rebuilt the hand-bumped constant with extra steps.

A ratchet also measures how many tests ran, not whether they were meaningful. It does not replace the check I wrote about in a merge gate must prove the tests ran, which asks whether the named tests executed at all. Use both: the count catches silent loss in bulk, and the named checks catch the one test that matters.

Checklist

  1. Derive the floor from the suite as it is today.
  2. Store the best passing result, moving up only.
  3. Update the best only from a run that passed.
  4. Make lowering the best a command with a reason, not a file edit.
  5. Print the floor and the reason in the failure message so nobody has to guess.

How this was verified

I ran the lab in a scratch directory. The table above is its output. Four unit tests passed: a literal floor passes a run that lost ten tests, the ratchet fails the same run even though the ratio rule alone would pass it, the best value never moves down, and the floor follows a growing id count. For a failing control I removed the best-run term from the ratchet; the catch test then failed, so the test can fail. I did not run this against a real continuous-integration system, and the 0.75 ratio is my example, not a recommendation.

CONNECTED READINGForward and back
Keep reading

The answer deserves
another experiment.

Back to the journal