FIELD NOTE / X
Ten tests are a start, not a finish line.
The complete written thought and the evidence behind it. The video edition will follow its public release.
The written argument is here.
This approved X edition is on the journal now. Its video player and original platform link will appear after each public release is verified.
Ten tests are a start, not a finish line.
Video caption
Ten tests are a start, not a finish line. Model, UI adapter or backend effect may be responsible. Do this because a fixed test count can create false certainty. #EricFieldNotes
Full written post / accessibility read
I like listing ten safe experiments before committing to an agent architecture. But the count is not the achievement. Each test should discriminate between two plausible explanations of a failure and tell us whether to proceed, change the harness or stop.
Imagine a computer-use run that submits twice after a timeout. Was the model repeating an action, did the adapter replay a call, or did the server process one request twice? A longer prompt cannot tell you which boundary failed.
Give each action and business request an ID. Simulate a lost acknowledgement while retaining the tool-call trace, host execution log and authoritative event ledger. The first divergence locates the mechanism and changes the repair.
After each result, ask what essential experiment would most change the release decision. Stop when remaining unknowns have an owner and an acceptable bound, or hold release. The loop is a method for convergence, not a ritual of ten green boxes.
#EricFieldNotes
Four-beat scene transcript
1. Ten tests are a start, not a finish line.
I like listing ten safe experiments before committing to an agent architecture. But the count is not the achievement. Each test should discriminate between two plausible explanations of a failure and tell us whether to proceed, change the harness or stop.
Visual: The next experiment should change a decision.
2. A retry failure can have three owners.
Imagine a computer-use run that submits twice after a timeout. Was the model repeating an action, did the adapter replay a call, or did the server process one request twice? A longer prompt cannot tell you which boundary failed.
Visual: Model, UI adapter or backend effect may be responsible.
3. Inject one fault at a time.
Give each action and business request an ID. Simulate a lost acknowledgement while retaining the tool-call trace, host execution log and authoritative event ledger. The first divergence locates the mechanism and changes the repair.
Visual: Use a synthetic run and compare three logs.
4. Continue until residual risk is explicit.
After each result, ask what essential experiment would most change the release decision. Stop when remaining unknowns have an owner and an acceptable bound, or hold release. The loop is a method for convergence, not a ritual of ten green boxes.
Visual: Do this because a fixed test count can create false certainty.
Research and claim limits
- OpenAI Computer use API guide (S159)
- OSWorld 2.1 official repository (S162)
- Playwright Tracing API (S163)
The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.