FIELD NOTE / TIKTOK
This computer-use demo looks finished.
The short film, the complete written thought, and the evidence behind it.
This computer-use demo looks finished.
Video caption
This computer-use demo looks finished. The UI and its test shared an optimistic view. Do this because a replay is not the outcome. #EricFieldNotes
Full written post / accessibility read
Watch a fictional test account. The agent changes a role, presses Save and gets the green toast. If this were the whole demo, I would believe it. Now close the session and read the authoritative role.
A delayed policy check rejects the requested role. The first browser session still rendered it briefly. This is an oracle design problem: the navigator and grader looked at the same temporary evidence surface.
The harness tags the request ID, waits for policy settlement, reads the audit state through a read-only route and logs in as the affected fictional user. Both disagree with the green screenshot, so the run is a failure.
Deliberately inject this delayed rejection in a resettable test tenant. If the suite still passes, repair the acceptance oracle before buying a stronger model. Better clicking will not fix a test that asks the wrong question.
#EricFieldNotes
Four-beat scene transcript
1. This computer-use demo looks finished.
Watch a fictional test account. The agent changes a role, presses Save and gets the green toast. If this were the whole demo, I would believe it. Now close the session and read the authoritative role.
Visual: The saved role disappears when we reopen it.
2. The agent did not necessarily click wrong.
A delayed policy check rejects the requested role. The first browser session still rendered it briefly. This is an oracle design problem: the navigator and grader looked at the same temporary evidence surface.
Visual: The UI and its test shared an optimistic view.
3. Show the correction path.
The harness tags the request ID, waits for policy settlement, reads the audit state through a read-only route and logs in as the affected fictional user. Both disagree with the green screenshot, so the run is a failure.
Visual: Retain the trace and check a separate record.
4. Make the false green turn red.
Deliberately inject this delayed rejection in a resettable test tenant. If the suite still passes, repair the acceptance oracle before buying a stronger model. Better clicking will not fix a test that asks the wrong question.
Visual: Do this because a replay is not the outcome.
Research and claim limits
- OpenAI Computer use API guide (S159)
- Playwright Tracing API (S163)
The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.