A full test suite can miss one required person.

Video caption

A full test suite can miss one required person.

Identity, state, surface and expected result.

Revoke after the session exists.

Coverage is about required states.

#EricFieldNotes

Full written post / accessibility read

Suppose a security decision requires a revoked admin to lose access immediately, including an already signed-in session. Tests cover fresh sign-ins and newly created users. Every observed result passes. The active-session cohort named by the decision never ran, so the claim of immediate revocation remains unverified.

Put newly signed-in, already signed-in and recently revoked identities in the protected matrix. For each, record the policy version, exact build, UI and API path, token age and expected denial. Do not turn the blank active-session row green because adjacent login tests passed.

In a disposable account, sign in, capture the session, then revoke the role and attempt the protected action through UI and API before that session naturally expires. Compare both reads with the source decision. If the fixture cannot create this state, the gate should say UNVERIFIED, not substitute a fresh login.

My rule: derive cohorts and state transitions from the source policy, then require one fresh result for each protected cell on the exact artifact. Do this because an agent can faithfully summarize every executed test while the only user who matters to the release was never represented.

#EricFieldNotes

Four-beat scene transcript

1. A full test suite can miss one required person.

Suppose a security decision requires a revoked admin to lose access immediately, including an already signed-in session. Tests cover fresh sign-ins and newly created users. Every observed result passes. The active-session cohort named by the decision never ran, so the claim of immediate revocation remains unverified.

Visual: The protected cohort is absent.

2. Make the missing row visible.

Put newly signed-in, already signed-in and recently revoked identities in the protected matrix. For each, record the policy version, exact build, UI and API path, token age and expected denial. Do not turn the blank active-session row green because adjacent login tests passed.

Visual: Identity, state, surface and expected result.

3. Exercise the absent cohort.

In a disposable account, sign in, capture the session, then revoke the role and attempt the protected action through UI and API before that session naturally expires. Compare both reads with the source decision. If the fixture cannot create this state, the gate should say UNVERIFIED, not substitute a fresh login.

Visual: Revoke after the session exists.

4. The matrix comes from the promise.

My rule: derive cohorts and state transitions from the source policy, then require one fresh result for each protected cell on the exact artifact. Do this because an agent can faithfully summarize every executed test while the only user who matters to the release was never represented.

Visual: Coverage is about required states.

Research and claim limits

• GitHub Docs: ruleset required status checks (S146)
• Argo Rollouts: analysis overview (S148)

The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.