FIELD NOTE / TIKTOK
The best coding model may be a poor browser tester.
The complete written thought and the evidence behind it. The video edition will follow its public release.
The written argument is here.
This approved TikTok edition is on the journal now. Its video player and original platform link will appear after each public release is verified.
The best coding model may be a poor browser tester.
Video caption
The best coding model may be a poor browser tester. A click trace is not a durable postcondition. Pay for accepted task outcomes. #EricFieldNotes
Full written post / accessibility read
A model ranks well on code patches. Now ask it to complete a sign-in flow and verify a changed account state in a browser. It may click the right button but miss a stale session or a failed backend save. Code success and computer-use success are separate contracts.
For a disposable account, record identity, initial state, action trace and an independent readback after reload. Time the complete accepted task, including retries and human rescue. Run the code task under its own protected tests. Do not average them into one impressive model score.
Use the same candidate models and budgets across repeatable code and browser cases. When the UI, model or tool wrapper changes, rerun the slice. Route a task only to paths that meet the required quality and permission boundary; put an owner escalation on uncertain results.
The rule: benchmark models separately on the work they will actually do, with independent acceptance and total correction cost. Do this because a brilliant patch model can still be the wrong browser operator, and a fast browser click can still leave the requested state unchanged.
#EricFieldNotes
Four-beat scene transcript
1. The best coding model may be a poor browser tester.
A model ranks well on code patches. Now ask it to complete a sign-in flow and verify a changed account state in a browser. It may click the right button but miss a stale session or a failed backend save. Code success and computer-use success are separate contracts.
Visual: Different tools create different failure modes.
2. Measure the full browser outcome.
For a disposable account, record identity, initial state, action trace and an independent readback after reload. Time the complete accepted task, including retries and human rescue. Run the code task under its own protected tests. Do not average them into one impressive model score.
Visual: A click trace is not a durable postcondition.
3. Select a route for each class.
Use the same candidate models and budgets across repeatable code and browser cases. When the UI, model or tool wrapper changes, rerun the slice. Route a task only to paths that meet the required quality and permission boundary; put an owner escalation on uncertain results.
Visual: Pin model, harness and version.
4. Do not buy one universal badge.
The rule: benchmark models separately on the work they will actually do, with independent acceptance and total correction cost. Do this because a brilliant patch model can still be the wrong browser operator, and a fast browser click can still leave the requested state unchanged.
Visual: Pay for accepted task outcomes.
Research and claim limits
- OSWorld 2.0 paper (S01)
- SWE-bench Live paper (S22)
- SWE-bench Pro Verified paper (S23)
The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.