Every green test should expose the next risk.

Video caption

Every green test should expose the next risk.

The browser passes; the fresh session fails.

A read-only record may beat another agent replay.

Do this because zero uncertainty is not a release plan.

#EricFieldNotes

Full written post / accessibility read

A checklist suggests the work ends at item ten. In an agent workflow, an early test often reveals a new failure boundary. The important question is whether the next experiment reduces uncertainty that affects the release decision.

A synthetic scheduling flow looks correct in the current browser session. A reopened session shows a missing setting. That observation moves the hypothesis from navigation accuracy to persistence and account scope. Repeating the same visual test adds little.

Write the competing explanations, the observation each predicts and the smallest safe probe that separates them. Run on a resettable fixture. Preserve failed traces and unexpected results, because they tell the next engineer what not to assume.

Record what passed, what remains untested, the consequence if wrong and the owner who accepts that residual risk. Then revisit after a model, UI or policy change. An experiment loop is useful only if its evidence updates a real decision.

#EricFieldNotes

Four-beat scene transcript

1. Every green test should expose the next risk.

A checklist suggests the work ends at item ten. In an agent workflow, an early test often reveals a new failure boundary. The important question is whether the next experiment reduces uncertainty that affects the release decision.

Visual: An experiment spiral can be more honest than a checklist.

2. One result changes the task model.

A synthetic scheduling flow looks correct in the current browser session. A reopened session shows a missing setting. That observation moves the hypothesis from navigation accuracy to persistence and account scope. Repeating the same visual test adds little.

Visual: The browser passes; the fresh session fails.

3. Choose the cheapest falsifier.

Write the competing explanations, the observation each predicts and the smallest safe probe that separates them. Run on a resettable fixture. Preserve failed traces and unexpected results, because they tell the next engineer what not to assume.

Visual: A read-only record may beat another agent replay.

4. Stop with a bounded residual risk.

Record what passed, what remains untested, the consequence if wrong and the owner who accepts that residual risk. Then revisit after a model, UI or policy change. An experiment loop is useful only if its evidence updates a real decision.

Visual: Do this because zero uncertainty is not a release plan.

Research and claim limits

• OpenAI Computer use API guide (S159)
• OSWorld 2.1 official repository (S162)

The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.