Build the test harness around cross-system invariants

The same policy implementation went from 16 of 42 independent mutations caught to 42 of 42 after I expanded its exercise. The source hashes did not change between those two receipts.

That is a useful result for an engineering leader. The improvement was in the questions the test system could answer. A passing implementation had not automatically supplied a complete acceptance argument. Independent faults exposed the missing observations, and the follow-up made those observations executable.

I have comparable retained cohorts for memory and task ownership. They are different systems with different fault sets, so I keep their numbers separate. Together they inform a practical architecture: a test harness should follow a promise across the components that can break it, with a state oracle independent of the actor and a containment boundary around the experiment.

This applies whether an engineer or an agent wrote the change. AI makes the need more visible because it can generate implementation, tests and a confident summary in the same session. The organization still needs evidence that the intended effect occurred under the intended authority.

Begin with the promise that crosses components

A unit test can correctly establish that one function returned a value. A product promise often crosses several functions, processes and stores before it becomes true.

Consider a queued action. A request enters the service, a job becomes claimable, a worker acquires ownership, an external effect occurs, a durable result is published and the user receives an answer. Every component can look locally healthy while the overall promise fails. The worker may report completion after losing its lease. The effect may occur while the receipt is lost. The answer may be delivered to an audience that changed during execution.

I begin the test contract with the outcome and identity. Which organization, principal, operation, target object and candidate version does this promise concern? What value should persist? What must never happen? What is the allowed observation delay? Which state remains uncertain if an acknowledgment disappears?

That contract tells me where a unit test is sufficient and where the harness needs to observe another system. It also gives QA a product-level responsibility: protect the meaning of the accepted outcome while implementations and model-generated plans change around it.

Separate the actor from the state oracle

An agent clicking Save can supply action evidence: it reached the control, submitted the form and observed the application's response. A success toast establishes what the interface displayed. It cannot establish that the intended record persisted for the intended user.

The state oracle reads the target independently. Depending on the boundary, that might be a service API, a database snapshot or a fresh cold application session after the allowed delay. The test names the object and expected value before the action, so the reader cannot accidentally confirm a nearby record or inherit the actor's mistaken target.

Independence is about the path of evidence. Another screenshot from the same warm browser can share the same stale client state. Another model reading the actor's summary can inherit the same interpretation. I want a read that can disagree with the actor, and a deliberate control that proves disagreement is detectable.

For a persisted preference, I would preserve the actor's success report, then independently read the exact user's stored preference. For a payment, I need the operation identity and ledger outcome, including whether the effect happened once. For permission changes, the decisive probe is a protected action evaluated under the changed authority. Each promise determines its own oracle.

I made that distinction executable in a four-case local demonstration. A separate actor process always reports saved. The correct case changes the intended record; three fault cases omit persistence, write the other organization or write the other object. A fresh read-only SQLite connection after the actor exits accepts only the correct case. It checks both the expected target and unchanged neighboring targets. This synthetic fixture proves the observer can disagree; it does not enforce a sandbox around the actor.

Put the acceptance contract outside the actor's edits

A test that follows an agent's rewritten requirement can pass while the original condition disappears. The code and test may agree perfectly about the wrong product behavior.

I retain the acceptance contract separately from the implementation attempt. It includes the original condition, the selected interpretation, the target identity and the forbidden effects. Reviewers can trace a test back to that source instead of trusting the author's latest summary of what was intended.

Protection has to match the actual execution model. A read-only file inside one worktree does not protect another writable copy, a mounted directory or a service endpoint that accepts updates from the same identity. The harness should name who can change fixtures, who can invoke the actor and who can issue the final verdict.

For the architecture in this essay, fixture integrity is an input to the verdict model. The model consumes the collector's verified fact; it does not implement filesystem permissions or authenticate the collector. That separation makes the missing infrastructure explicit and gives its owner a testable obligation.

Inject faults at the handoff, not just the function

The useful fault matrix follows the promise through its boundaries.

Before dispatch, change the audience or revoke the capability. During dispatch, lose the reply after the receiver accepted the operation. During execution, let a lease expire while a worker still runs. Before publication, change the generation or epoch. After publication, restart the reader and inspect whether the result and its identity survived.

Those faults exercise different promises. A permission test asks whether the effect is refused under current authority. A lost-reply test asks whether the operation becomes uncertain without being blindly repeated. A publication test asks whether a stale worker can make its output authoritative. A persistence test asks what a fresh reader sees after the writer's session ends.

I do not call a matrix executed because the rows are written down. Each row needs a reproducible setup, the injected change, the decisive observation and the actual result. Some require real database or network integration. Others can be qualified as narrow local contracts. Keeping those scopes visible is part of the harness, not editorial decoration.

What the independent cohorts actually showed

The retained memory exercise initially caught 17 of 30 independent mutations. After the missing cases were added, the same set produced 30 of 30 with a clean control. The source bytes stayed the same within that cohort.

The policy exercise initially caught 16 of 42. Its same-set follow-up caught 42 of 42 with a clean control and unchanged policy source hashes. The added cases covered concrete obligations such as a signed grammar digest mismatch, a control returning bytes alongside an error and limits that the initial exercise did not expose.

The local task-ownership exercise initially caught 23 of 33 non-equivalent mutations. Its expanded exercise caught the same 33 of 33, while one expected-equivalent mutation survived. That survivor was retained as equivalent, rather than creating another test solely to force a perfect-looking count.

These are exercise-sensitivity results, not production defect rates. The follow-ups reuse known faults, so they demonstrate closure of those gaps rather than a fresh independent challenge. I retain the original scores beside them because the first pass shows what the original mental model missed. A new blind set would be needed to investigate the remaining unknowns.

Make a failed run explain what it established

A process exiting nonzero is evidence that the process did not complete successfully. It does not, alone, establish that a test detected the intended defect.

A mutant might fail to compile. Its fixture may be missing. The unmodified control may already fail. The state oracle may be unavailable. A dependency may hang before the relevant action. Counting every one of those as a caught semantic defect turns an unreliable test environment into an apparently excellent suite.

I wrote a small teaching model to make those outcomes explicit. It receives observations from a trusted collector: source and exercise hashes, fixture verification, compile state, timeout state, process exit and oracle verdict. Before scoring the mutant, it validates the contract, binds the observations to the expected identities and requires a healthy unmodified control.

That model is a decision contract for a future reusable harness. It gives the collector and executor precise output requirements. It is not a claim that the existing review scripts already provide every one of those assurances.

Keep semantic failures and liveness failures distinct

An independent oracle can prove a semantic violation even when the actor exits zero. The actor reported success, but the stored value or protected effect contradicts the invariant. That is precisely the disagreement I want the harness to preserve.

A timeout needs a different argument. If the invariant is responsiveness, the observation window was part of the contract and a trusted probe established that the required response missed that deadline, a timeout can be a meaningful liveness failure. A generic runner deadline without that attribution is inconclusive.

The teaching model expresses the distinction directly:

    if mutant.timed_out is not False:
        if (mutant.timed_out is True and invariant_kind == 'responsiveness'
                and mutant.oracle == 'deadline_violation'):
            return 'killed_liveness'
        return 'inconclusive_timeout'
    if mutant.oracle == 'violation':
        return 'killed_semantic'
    if mutant.oracle != 'pass':
        return 'inconclusive_oracle'
    if mutant.exit_code != 0:
        return 'inconclusive_execution'
    return 'survived'

This keeps a nonzero process exit from substituting for an oracle. It also allows an independently observed semantic failure to matter even when the actor's own status is green. The collector must still authenticate, bind and attribute those observations in the real system.

Qualify the control before trusting the mutation

The unmodified control establishes whether this exercise can evaluate the original implementation in this environment. If it cannot, the mutant result has no stable comparison.

I want the control to compile, complete within its contract, exit successfully and satisfy the oracle. Its source and exercise identities must match the intended run. The protected fixture must be the expected one. Only then does a changed source observation become eligible for a sensitivity judgment.

The model's 66-case exercise varies those dimensions independently. It checks wrong source hashes, wrong exercise hashes, malformed digest contracts, unverified fixtures, broken controls, invalid mutants, unknown oracle outcomes and the distinction between semantic and responsiveness failures. Three syntax-valid author faults make the exercise fail assertions; restoring the original model returns zero.

Those controls qualify the cases I designed. They do not replace independent review of the model. I keep the author controls, the independent mutation set and any later same-set replay as distinct records, so the provenance of each result remains visible.

Protect the experiment from its own faults

A mutant intentionally changes behavior. It can remove a timeout, create an infinite loop, start another process or redirect a write. A test harness that safely runs the original program may not safely contain those altered behaviors.

I treat containment as an outer execution contract. The runner should have an owned disposable filesystem, bounded process and memory resources, controlled network destinations, scoped database credentials and a cleanup mechanism that survives the child it is cleaning up. The protected oracle and verdict store should not become writable simply because the actor can execute code.

A Python subprocess with a timeout is not a complete sandbox. An isolated directory is not a network policy. Killing the immediate process does not necessarily settle what its children or remote actions did. The executor must establish its actual boundaries, and the test must probe the routes it relies on.

The work here does not run arbitrary hostile mutants or modify production state. Its executable artifact is the trusted-observation verdict model. The outer containment design is an explicit obligation for an integrated runner, with harmless boundary canaries required before any broader assurance.

Retain identity through the evidence chain

The test result needs to identify what ran, which fixture it used and what the observer checked. A command name and a green badge are insufficient once several agents, worktrees or providers are active at the same time.

I retain the exact source hash, exercise hash, mutation-set identity, control outcome, mutant label and result class. For a cross-system test, I also want organization, operation and target identity, the observation window, oracle path and the state version that was read. Sensitive values need their own capture policy; diagnostic linkage does not require dumping full prompts or customer records.

The record should distinguish an assertion, a compile failure, a timeout and an unavailable observation. It should also preserve expected-equivalent mutants separately. Otherwise a later reader cannot reconstruct the denominator or tell whether an impressive score counted invalid experiments.

The same discipline applies to fixes. A same-set follow-up can demonstrate that known gaps are closed. It should retain the original receipt and identify exactly which source or exercise changed. Relabeling the follow-up as a new independent result would make the evidence less useful to the next engineering decision.

Give QA ownership of outcomes and uncertainty

This changes the shape of QA work. There is still value in writing unit tests and exploring the interface. The higher-leverage responsibility is defining the product promise, selecting independent observations and deciding which unknown states prevent acceptance.

For a user-facing change, QA can own the target identity and cold readback. For security-sensitive work, the owner can define the forbidden effect and prove the control catches a harmless attempted violation. For release automation, the owner can distinguish candidate eligibility, actual execution and the accepted result.

That role needs engineering judgment. Someone must decide whether a timeout tests responsiveness or merely exposes a slow fixture, whether a surviving mutation is equivalent and whether the observer shares the actor's stale state. More generated tests do not remove those decisions. A well-designed harness makes them easier to examine and repeat.

For CTOs planning capacity, I would fund that ownership before scaling parallel implementation. Otherwise every additional agent creates more plausible results waiting for the same small group of people to determine what they actually prove.

Build one invariant path before expanding coverage

Choose one promise that matters to a customer or an operator. Write its target identity, expected durable effect, forbidden effect, observation deadline and uncertainty owner.

Run the normal action and obtain an independent readback. Then inject a harmless disagreement: leave the visible success path intact while preventing the expected durable change in an owned fixture. The oracle must reject the outcome. Add a broken-control case so environmental failure cannot masquerade as sensitivity. Add a lost-result case and verify the workflow preserves uncertainty rather than inventing success or a fresh operation.

Only then expand the matrix to more surfaces and fault boundaries. This produces a useful sequence of accepted capabilities: one promise, one oracle, one failure control and a retained result. It also exposes which integration owner must supply the next missing observation.

I use the independent cohorts as a reminder of why this matters. The source can stay unchanged while the harness learns to ask better questions. The investment is a reusable ability to distinguish accepted outcomes, real violations and work that remains unverified.

How this was verified

The historical cohort table is derived from six retained review receipts at content-video commit 30d2cb6: memory 17/30 then same-set 30/30; policy 16/42 then 42/42; ownership 23/33 then 33/33 plus one expected equivalent. Each pair has matching source hashes and clean controls. The historical review runner and raw receipts remain in the packet. Its exit-based killed labels are retained as historical classifications; they are not retrospectively upgraded to the stronger typed assurance model proposed here.

The new verdict model was committed at 4d1c480 before its author exercise. The 66-case exercise and three author fault controls execute compiled source bytes, avoiding mutable import-cache effects. A separate four-case synthetic persistence demo uses a child actor process and fresh read-only SQLite connection; all actors report saved, but only the correct target/value with unchanged neighboring targets is accepted. The verdict model’s initial independent review caught 25 of 25 mutations with control exit zero. The set was pinned before the reviewer read the exercise; this is a blind source-based review, without claiming its set commit preceded the author exercise commit. The receipt binds the unchanged model and 66-case exercise hashes. The verdict artifact consumes trusted collector facts; it does not authenticate receipts, implement the oracle, protect fixtures, contain arbitrary processes or prove a deployed cross-system workflow. The broader harness is an architecture synthesis with those obligations left explicit.