A merge gate must prove the tests ran

A crashed test worker reported its tests as skipped, not failed. My gate would have treated that run as evidence of safety.

I was building an automated merge gate around the idea that a green test command is a meaningful signal. The first implementation tried to infer that signal from test counts. It kept refusing legitimate commands like lint, formatting and static analysis because those tools do not print a test total. That pushed me toward a deceptively simple fix: make the parser accept more output shapes.

The deeper issue was not output formatting. The gate needed to answer two independent questions: did the command succeed, and did the tests we intended actually execute? A zero exit code alone is enough for a known non-test check. It is not enough for a test suite if the worker crashed, skipped every case, or never loaded the file.

Evidence has to match the check

We now classify commands by their contract. For a known lint, format, typecheck or ancestry check, exit status is the result. For a test command, the gate also requires credible execution evidence. An empty success report is a stop. A test file that never ran is not a passing file. A worker crash that reports the remaining tests as skipped is not a clean suite.

That required parsing the formats we actually run rather than assuming every tool emits a single count. The parser handles nested test identities and failure records; it does not treat arbitrary text that resembles a test line inside a diagnostic block as an executed assertion. When a test process exits without a trustworthy test plan, the result is unknown and the merge waits.

A test added by the change creates another edge. It cannot run against the unmodified main tree because the file is not there. So the gate does not invent a baseline for that file. It evaluates the new test with the merged tree and still compares the existing tests against the live baseline.

Compare new failures, not an impossible perfect green

A strict “both sides must be fully green” rule looked safe until we measured the current main branch. Existing failures would have stopped every release. The practical contract is narrower and more useful: run the same relevant suite on the live baseline and on the candidate; require no new failing test identities; and stop if the baseline itself is missing or the two runs do not have the same shape.

That comparison is not permission to ignore old failures. It isolates regression responsibility while preserving a visible baseline that can be improved separately. The gate records the existing failures, compares identities rather than just totals, and treats a missing or incompatible baseline as unavailable evidence.

The difference matters. If the count changes from 10 to 10, a simple total says nothing about whether one old failure disappeared and a new one appeared. Identity-based comparison can show that. If the suite ran 10 tests on one side and silently skipped half on the other, matching counts also cannot establish equivalence. Execution status and failure identity belong in the evidence record.

A rerun is not a pass

We also had to decide what to do with flaky results. Re-running the whole command once can help classify a failure, but a green rerun cannot erase the first failure. The outcome is flaky or unresolved, not pass. If a failure reproduces, it remains a failure. If it does not, the record keeps both runs and the merge stays held for a decision under the project’s policy.

This prevents the gate from laundering uncertainty into a green check. A retry is more evidence, but it does not rewrite what happened on the first attempt.

The evidence object is part of the security boundary

A useful test result needs identity: source tree, command, environment, start and end, exit status, test execution evidence, failures and whether a baseline was reused. The result should bind to the exact candidate tree, not merely a branch name or commit label that may point somewhere else by the time it is consumed.

The same principle applies to cache hits. A cached result is acceptable only when its key represents the relevant source and execution inputs. If the cache cannot establish that match, it is a miss. A stale green answer is worse than a slow rerun because it creates confidence without work.

I now think of the parser as part of the release gate, not a convenience layer around it. It decides whether the system has proof at all. Its states need to include PASS, FAIL and STOP/UNKNOWN; treating every parse problem as either pass or ordinary failure hides the distinction between known bad and not known.

A practical gate contract

For each command, declare whether it is a test or a check. Bind the run to the exact candidate tree. For tests, prove that the intended cases executed and report failures by stable identity. Compare against a current, matching baseline, count only new failures as regressions, and stop when baseline evidence is absent. Preserve the first failure across retries. Keep cached evidence only when its complete key matches. Finally, make “unknown” block the merge until an owner resolves it.

This took more care than adding another regex to accept another tool’s output. But that is the right level of engineering: a merge gate is deciding what the organization is willing to call verified. What does your pipeline currently record when the command exits zero but the test file never ran?

Bind execution evidence to one attempt

I keep the evidence record about the run, not just the command string. It needs the target revision, command classification, environment, start and finish, exit status, and the test identities that actually executed. A static check may be judged by its exit status. A test run also needs positive evidence that the intended cases loaded and reached a terminal state. Skipped, cancelled, crashed or never-collected tests cannot be silently counted as green.

The parser should be conservative when output is unfamiliar. It can recognize the formats we run, retain the raw report, and return an explicit unknown outcome for anything else. Broadening the parser until every output looks successful is the wrong goal. The goal is to make a human able to trace a result back to a specific command and commit, while making omissions visible.

Bind execution evidence to one attempt

A useful run record needs more than a command line and a green exit code. I bind the result to the source revision, test selection, runner identity, start and finish state, and the worker’s final disposition. A retry is a new attempt with a new identity; it must not fill a missing result in the earlier attempt. This matters when an orchestrator restarts after its child has already begun work. Without that distinction, a second run can make the overall job look healthy while the first run’s partial result disappears.

I also separate “the runner could not determine the outcome” from “a test failed.” A missing result is not evidence that the test passed, but it is also not necessarily a product defect. Keeping UNKNOWN visible directs the next action: recover a lost worker record, rerun the exact selection, or escalate a broken harness. A single boolean loses that operational choice.

For a release gate, I want a durable attempt record that can be checked without trusting the process that produced it. The record should say what was expected to run, what actually started, which cases completed, and whether any were skipped or cancelled. The gate compares those sets. If the expected set and completed set differ, the output is incomplete regardless of the summary string. The review then has a concrete gap to resolve instead of a vague “tests were green.”

I also ask what happens if a worker dies between reporting progress and returning its final summary. The runner needs to reconcile durable child results with the parent process status; a parent crash cannot convert unfinished cases into skips. What exact evidence would your gate retain if the test worker vanished halfway through a suite?

How this was verified

I checked the source and receipts listed in this package’s source packet. Measurements are presented only where a durable receipt supports them; design implications are identified as operator conclusions.