The release gate is the product

Field note for Week 8 · Days 50–56 · Eric MacDougall

The easiest way to misunderstand agentic QA is to ask another agent to look at the first agent's patch and call that “independent.” The agents may share the same shortened requirement, the same test fixtures, the same editable workspace and the same pressure to produce a green result. A second opinion can be useful. It is not automatically an external measurement.

As agents write more of the code, the team's most important software becomes the path by which code earns permission to affect customers. That path has to preserve the original product decision, attach fresh evidence to the exact release artifact, distinguish failed from missing observations, and hold authority for exceptions. It also has to continue after deployment, where the real customer cohort can reveal behavior the test environment did not contain.

This is why I call the release gate the product. The gate determines what your organization is willing to call “working.”

A green suite can implement the wrong agreement

Consider a fictional SaaS team with an enterprise customer exception: one tenant may use a feature during a migration, while others need a paid entitlement. A product manager records that exception in a customer decision. Three agents then work in parallel on API, UI and migration code. The task summary sent to the agents says “make the feature available to enterprise tenants.” All three write coherent code and tests. CI is green. The interface works. The original migration-only condition is gone.

This is not a simple missing unit test. The tests are accurately verifying the shortened instruction the agents received. The failure is in the decision-to-acceptance chain. A reviewer who sees only the patch and generated tests can miss it too. The source of authority was the customer decision, but the implementation and evidence have drifted away from it.

The fix starts before code generation. Assign a decision owner. Record the original source, affected cohort, permitted behavior, forbidden behavior, expiry and the observation that would reverse the decision. A worker can propose a change to that contract, but must not silently edit the contract it will be graded against.

Three evidence rails

For consequential work, I want a release packet with three independent rails:

These rails cannot be collapsed into one generated score. A test suite can pass while the product target is wrong. A screenshot can show a click path while the external record is wrong. A canary can look quiet because the affected cohort is absent. Each rail needs its own source, owner and threshold.

GitHub rulesets can require named checks and an expected GitHub App. Its documentation also notes that write-capable people or integrations can otherwise set a status check state. That is an authority lesson: a green badge should come from a trusted grader, not merely from a job the coding worker could replace. It is still only a badge for the claims that job actually tested.

What a real QA agent can do

A QA agent is valuable when it broadens the adversarial search. Give it the original product decision and the proposed artifact separately. Ask it to find missing cohorts, state transitions, timeouts, migration paths, permissions and contradictory UI/API behavior. Let it generate candidate probes, observe browser or desktop flows, attach traces, and explain where evidence ends.

Then run those probes under a grader outside the worker's editable tree or permission scope. Bind the result to the exact commit and build digest. Require non-empty, fresh test reports. Make a deliberate negative control fail: remove the entitlement check in a disposable mutant and prove the protected test turns red. If that mutant passes, the suite does not support the entitlement claim, regardless of how many test names sound relevant.

Playwright traces are useful because they show actions, snapshots, console and network activity. The direct tracing API documentation specifically distinguishes captured browser operations from test assertions. A trace helps investigate what happened; it cannot independently establish that the customer's entitlement rule was correct. Check the postcondition in a separate read of the real application state and make the expected tenant, user and artifact explicit.

Computer-use models add another observation surface. They can run realistic flows and expose unexpected UI behavior. They can also misread visual state, leave a session signed in, or confuse an accepted request with a durable result. Record environment, identity, action trace and independent outcome. A visual pass without a target-state check is incomplete evidence.

Fail, pass and unverified are different paths

Many agentic pipelines have two colors: green and red. They need a third state: unverified.

If a test did not run, its report is stale, the browser session was already warm, the canary has too few relevant customers, or a required external state cannot be read, the system does not know the outcome. A missing report should never be interpreted as “zero failures.” A model's summary of an omitted test is not a substitute. Put the uncertainty in front of the release owner and pause the gate.

Argo Rollouts embodies a useful distinction: successful analysis can continue a rollout, failed analysis can abort it, and inconclusive analysis can pause it for judgment. That is a technical primitive, not a complete product policy. The team still has to decide what metric, cohort, duration and business harm threshold make the analysis meaningful.

The same three-state pattern belongs in local CI, browser QA and product signoff. A legitimate exception may override an unverified gate in some low-risk context, but the override must name an owner, evidence gap, scope, expiry, compensation and rollback. Hidden manual bypasses destroy the feedback loop.

QA continues after merge

The product only meets real traffic after release. Google's SRE guidance on canarying explains why a small, measured exposure helps find production-only behavior with limited impact. The canary's signal must be separated from the control. If the special entitlement cohort is only one tenant, aggregate error rate across thousands of unaffected tenants can hide the exact failure the gate was meant to catch.

For the fictional SaaS exception, I would deploy the exact artifact to a bounded cohort. Then I would check three questions: did the migration tenant get the promised access, did a non-entitled tenant remain blocked, and did the customer-facing workflow complete without manual rescue? A generic HTTP 200 rate is useful but insufficient. If the cohort is too small to conclude, the answer is inconclusive. Pause or use a defined manual review. Do not promote the release because the dashboard had no red pixels.

NIST's Secure Software Development Framework recommends security practices throughout each SDLC, including verification and attention to root causes. For agentic teams, one root cause is recurring drift between the decision a business actually made and the narrower task the worker tested. Fix that handoff and the protected acceptance fixture; otherwise each incident becomes another one-off prompt correction.

Ten experiments before trusting the gate

These are proposed experiments for a disposable environment. None have been run against Eric's live product:

Local synthetic fixture. I ran eight deterministic checks against a fictional release-packet companion. All eight passed. The useful negative control put a nominal PASS report on a different artifact digest; the local gate classified that packet UNVERIFIED. This tests the companion's decision logic for a wrong-artifact report. It does not authenticate the evidence issuer, verify a protected CI run, or prove that a real rollout target stayed unchanged. Those claims still need independent origin and external state readback.

1. Change the agent's test file to always pass. A trusted check from a separate source must still reject an entitlement mutant.
2. Run the correct suite against the wrong commit. The release packet must reject the digest mismatch.
3. Attach yesterday's browser report to today's build. The gate must reject stale evidence.
4. Empty the required test report while leaving the command exit code at zero. The outcome must be unverified.
5. Delete the migration-only clause from a worker summary but preserve the original decision. Independent acceptance must catch the broadened entitlement.
6. Make UI access look right while the API allows a non-entitled tenant. The postcondition check must reject the release.
7. Use an existing signed-in session for a test that requires a new user. The clean-session probe must fail or remain unverified.
8. Hide the affected tenant from the canary cohort. The canary must pause for insufficient relevant exposure, not report product success.
9. Inject a metric-provider outage during rollout analysis. The pipeline must pause and page a named owner.
10. Record an override, let it expire, then rerun the gate. The release must require renewed evidence or a newly authorized decision.

The point is not to maximize test count. It is to make claims falsifiable under the runtime and authority boundaries that actually ship the product.

How I would operate the team

Engineers still need fundamentals. They now spend more of their time deciding what is true, what is permitted, what must be tested, and who owns the exception. Agents can write substantial code, propose tests, inspect traces and carry out narrow remediation loops. Product owners must retain the customer intent. QA must retain its right to say “not verified.” Release operators must retain the ability to pause and reverse exposure.

Measure accepted outcomes and their rework: how often a release packet matched the decision, how many changes needed reprompting and retesting, how long an exception remained open, how often a canary paused, what escaped, and how quickly the team restored the customer promise. Counting agent sessions, generated lines or green test names rewards activity without proving the product.

My rule: bind every release to a source-linked product decision, an exact artifact, independent behavioral evidence and a bounded operational readback. Promote only when all required claims pass; pause when evidence is missing; record the person who overrides. Do this because an agent team can produce flawless evidence for the wrong question at extraordinary speed.

Sources and limits

• GitHub rulesets and required status checks (S146).
• Playwright Trace Viewer and Tracing API (S147).
• Argo Rollouts analysis (S148).
• Google SRE Workbook: Canarying Releases (S149).
• NIST SP 800-218 Secure Software Development Framework (S150).

The enterprise-entitlement case is fictional. The ten tests are proposals, not measured outcomes. Check current product/tool behavior, customer policy and release topology before adopting the example.

The field kit

The companion folder for this week holds runnable examples, decision records, evidence and limits. The repository release log is updated after each article's live readback.