FIELD NOTE / X
Policy-change drills belong in agent QA.
The complete written thought and the evidence behind it. The video edition will follow its public release.
The written argument is here.
This approved X edition is on the journal now. Its video player and original platform link will appear after each public release is verified.
Policy-change drills belong in agent QA.
Video caption
Policy-change drills belong in agent QA. Structural validation does not prove business disposition. Require a passing continuity contract before release. #EricFieldNotes
Full written post / accessibility read
Your code can stay fixed while a model route refuses, loses access or returns an unfamiliar error. If QA only tests happy prompts, the first real route change tests your business process in production.
A structured output can be perfectly valid while a refused case is mapped to the default category. Token masking can enforce the shape of an answer; it cannot certify that the case should close. A downstream state readback is still necessary.
In a disposable queue, simulate an explicit refusal, a network cut after dispatch, an unrecognized provider code and a delayed result after the case changed. Verify owner, retry bound, no silent close and no action on stale data. Keep one expected-state oracle outside the worker.
Version the route configuration and run fault drills when it changes. Do this because a provider-policy boundary can change what your application receives, while your application still owns its downstream effects.
#EricFieldNotes
Four-beat scene transcript
1. Policy-change drills belong in agent QA.
Your code can stay fixed while a model route refuses, loses access or returns an unfamiliar error. If QA only tests happy prompts, the first real route change tests your business process in production.
Visual: A route can change without a code commit.
2. A schema can hide a wrong default.
A structured output can be perfectly valid while a refused case is mapped to the default category. Token masking can enforce the shape of an answer; it cannot certify that the case should close. A downstream state readback is still necessary.
Visual: Structural validation does not prove business disposition.
3. Inject four route faults.
In a disposable queue, simulate an explicit refusal, a network cut after dispatch, an unrecognized provider code and a delayed result after the case changed. Verify owner, retry bound, no silent close and no action on stale data. Keep one expected-state oracle outside the worker.
Visual: Refusal, timeout, unknown code and stale result.
4. Treat external rule changes like code changes.
Version the route configuration and run fault drills when it changes. Do this because a provider-policy boundary can change what your application receives, while your application still owns its downstream effects.
Visual: Require a passing continuity contract before release.
Research and claim limits
- OpenAI Usage Policies (S133)
- Anthropic Usage Policy update (S134)
- OpenAI structured outputs explanation (S08)
The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.