FIELD NOTE / X
A green test may be unable to fail.
The short film, the complete written thought, and the evidence behind it.
The X conversation link will follow its public release.
A green test may be unable to fail.
Video caption
A green test may be unable to fail. The gate must reject a forbidden production receipt. Deliberately make the forbidden state and watch acceptance fail. Narration uses Eric's authorized AI voice clone. #EricFieldNotes
Full written post / accessibility read
An agent's test asserts the tool returned blocked. The production tenant changes through another route. The assertion stays green because it never looked at the target. That is a vacuous compliance test, regardless of whether a junior developer or a strong model wrote it.
In a disposable service, add a test-only mutant that writes a production receipt despite a staged caller. Run the existing gate. If it passes, it cannot support the claim that production stayed unchanged. No real customer or production credential is involved.
First assert a recorded deny and zero new production receipts. Then compare the protected tenant's state hash before and after. A second mutant hides the receipt but changes state; the independent readback must catch it. Also run an authorized staging change to prove the sensor observes effects.
Treat the negative control as part of the harness contract. If it stays green, fix the oracle before adding more tests. Do it because a large test count can give false confidence when the checks never discriminate the consequence that matters.
Narration uses Eric's authorized AI voice clone.
#EricFieldNotes
Four-beat scene transcript
1. A green test may be unable to fail.
An agent's test asserts the tool returned blocked. The production tenant changes through another route. The assertion stays green because it never looked at the target. That is a vacuous compliance test, regardless of whether a junior developer or a strong model wrote it.
Visual: Checking tool text does not check forbidden effect.
2. Create the bad outcome safely.
In a disposable service, add a test-only mutant that writes a production receipt despite a staged caller. Run the existing gate. If it passes, it cannot support the claim that production stayed unchanged. No real customer or production credential is involved.
Visual: The gate must reject a forbidden production receipt.
3. Check two independent observations.
First assert a recorded deny and zero new production receipts. Then compare the protected tenant's state hash before and after. A second mutant hides the receipt but changes state; the independent readback must catch it. Also run an authorized staging change to prove the sensor observes effects.
Visual: Receipt stream and current state catch different misses.
4. Test the test.
Treat the negative control as part of the harness contract. If it stays green, fix the oracle before adding more tests. Do it because a large test count can give false confidence when the checks never discriminate the consequence that matters.
Visual: Deliberately make the forbidden state and watch acceptance fail.
Research and claim limits
The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.