Eric MacDougall Follow the work
All field notesFIELD NOTE / 009

Computer use / Days 57–60

What computer-use agents can prove.

A field guide to browser, desktop, and app agents: speed, blind spots, and outcome-level testing.

Eric MacDougall13 minute read · 2026-09-25
EVIDENCE / DECISION
An agent's work passes through independent controls before a release decision
FIELD NOTE 09 / COMPUTER USE
THE OVERVIEW FILMWatch the full argument

Open on YouTube ↗

Read the complete film transcript

What computer-use agents can prove — complete spoken transcript

Published transcript of the approved overview film. Performance cues are omitted because they are pauses or delivery directions, not spoken words. Timed English captions are available on the public YouTube player.

A computer-use agent can navigate your browser, desktop or app and hand you a convincing film of the work. That film can be useful. It can also be the most persuasive false green light in your QA process. A screenshot tells me what appeared on one surface, at one moment, to one session. If the task changes permissions, money, a schedule or a customer record, I need to know whether the intended outcome persisted, who was affected and whether the action can be safely reversed. The model's final answer is not that evidence.

Current computer-use systems differ in tools and surfaces. OpenAI describes code execution or structured computer actions. Anthropic distinguishes desktop computer use from webpage browser use. Google's Gemini loop sends actions to a client that executes them and returns a new screenshot. The common engineering point: your application owns the environment, credentials, action handler and acceptance gate. A more capable model can choose better clicks. It does not take responsibility for the authority you handed those clicks or for an outcome your harness never checked.

Consider a fictional access-request test. In an isolated tenant, the agent changes an employee's application role and sees “Saved.” The browser immediately shows the optimistic new value. But the server's policy engine rejects that role after the write attempt; a background update restores the old role. The agent captured the green toast and stopped. This is not a tale about a foolish agent missing a button. The UI and the model shared the same temporary view. If your oracle is the agent watching its own action, you have measured navigation and local rendering. You have not measured durable authorization.

I would split this task into claims before running it. What change was authorized? Did the agent use only allowed screens and accounts? What state did the system persist? What can the affected user actually do in a new session? Keep the trace and screenshots; they explain how a failure happened. Then read the authoritative record or audit event through a separate, read-only channel. Reopen the app with a fresh session and the affected role. If the job includes a notification, check its delivery state and duplicates. Each check has a different owner. A green toast cannot silently stand in for all four.

Before I trust the test, I poison it on purpose. Seed one valid request and one forbidden role. Make the fake policy service reject the forbidden case after the UI says “Saved.” If the suite still reports success, the acceptance test is coupled to the same optimistic view as the agent. Then inject a stale screenshot, a timed-out save, a duplicate notification and a fresh-session logout. These are safe synthetic experiments, not attempts on live accounts. The strongest demonstration is a test that turns red when the actual business result is wrong, even while the replay looks beautiful.

Browser DOM automation is often faster and easier to assert when the interface exposes stable structure. Screenshot-driven computer use earns its place when work crosses desktop apps, native dialogs, canvas views or awkward sites. Mobile adds gestures, permission prompts and device state. I would not force one agent to be both navigator and judge. Use the cheapest deterministic probe for stable fields and permissions. Use a visual agent where the real user journey requires perception and adaptation. Reserve independent readback for consequential state. Measure model plus tool adapter plus environment, because their errors combine.

Benchmarks help compare progress, but the release version, task assets and environment matter. OSWorld now recommends a pinned 2.1 release precisely because an evaluation is more than a model name. Your own workflow has roles, latency, intermittent UI, retries and exceptions that a public task set may not contain. Do not copy a leaderboard score into a production authorization. Build a small local corpus: accepted tasks, failed tasks, rescues, time per accepted outcome and reversals. Preserve the unsuccessful runs. A polished demo is a numerator without a denominator.

My release receipt has eight gates. Record the request and who may authorize it. Constrain account, sites, actions and time. Seed a known start state. Keep the action trace. Check an independent postcondition. Reopen in a fresh session. Run a negative control. Name who accepts, monitors and reverses the result. Some low-risk visual checks need fewer gates; high-consequence writes may need a human checkpoint. The point is to decide the evidence before the agent begins, so a persuasive final screenshot cannot renegotiate what success means.

Computer-use agents are valuable because they can test and operate the messy interfaces people actually use. They are weak when we confuse a recorded journey with a verified business result. My rule: let the agent navigate; make the harness prove the outcome. Define the authorized end state in advance, capture the trace, verify consequential effects from an independent surface, and count failures and rescue hours alongside successes. Expand autonomy only when that full acceptance rate holds on your own tasks. A convincing video is evidence of a journey. It is not, by itself, evidence that the system did the right thing.

What computer-use agents can prove

Field note 09 · computer use, UI testing and outcome evidence · Eric MacDougall

Overview film: the approved film and its timed English captions are public on YouTube and embedded with this article. Companion: the public Week 09 field-notes kit contains the synthetic acceptance receipt, negative controls and ten proposed probes. Related social links appear only after their public readback.

I like computer-use agents. They work at the level people work: a browser form, a desktop dialog, an app screen, an interruption. They can explore an interface when a brittle test script would need a week of selectors. They can uncover pain that unit tests never see. That makes them particularly useful for QA of the user journey and for controlled, repetitive operations.

Their strength creates a dangerous category error. A film of the agent clicking through a flow is persuasive. A final screenshot with a green toast can feel like proof that the work is done. For a consequential workflow, it may show only that the interface looked done in the same session the agent was operating. It may say nothing about durable state, affected users, delayed policy checks, duplicate side effects or how the system behaves after a fresh login.

My position is simple: let the agent navigate; make the harness prove the outcome. The harness must know what was authorized, which actions were in scope, what independent result would count as success, and which negative case must fail. That is an engineering choice, not a better prompt.

What current systems actually give you

The word “computer-use model” hides several architectural choices. OpenAI's current guide describes a model operating browser and desktop interfaces through either code execution or structured computer actions. The application supplies and keeps the execution environment, runs requested actions and returns screenshots or tool results. API conversation state and browser/runtime state are separate. Anthropic's computer tool documentation similarly places tool execution in the application; it distinguishes full desktop computer use from its webpage-focused browser tool. Google's Gemini Computer Use guide describes a request/action/screenshot loop in which client code scales coordinates, executes calls and sends back the new screen. Google currently labels that capability Preview and asks for close supervision on important tasks.

This is not a ranking. Tool contracts, model behavior, host adapters and product versions change. A model may be good at deciding the next action and poor at a particular visual state; a host adapter may click the wrong coordinate; an app may lie to both with optimistic rendering. Measure the whole system on the exact task and environment. The model does not own your credentials, action allowlist, timeout, isolation or acceptance rule. Your host does.

I divide computer-use work into four claims:

Claim Useful evidence What it does not settle
The agent followed a permitted path Tool calls, action trace, allowed URLs/accounts, screenshots Whether the server kept the requested change.
The interface rendered the expected state DOM assertion or independent visual comparison on a known device Whether the state is durable or visible to other roles.
The system committed the intended state Read-only authoritative record, API/audit readback tied to a request ID Whether the actual user can exercise it or a side effect duplicated.
The user-level outcome occurred Fresh-session role-based check, notification/delivery state, downstream observation Future reliability under different tenants, timing or UI variants.

These lanes are not a demand for a backend oracle on every pixel test. If the claim is “the layout does not overflow at a phone width,” the visual surface is the outcome. If the claim is “this employee now has access,” a screenshot of a green form is an intermediate observation. The acceptance evidence should match the claim and the cost of being wrong.

A fictional case that looks finished

Imagine an isolated test tenant with a browser-based access-request form. A computer-use agent is asked to change a fictional employee's role in an internal application. It opens the correct account, selects the new role, presses Save and sees a green “Saved” toast. The page immediately reflects the selection. The agent's trace is clean, and its final answer says the role was updated.

The application uses optimistic UI. A policy check runs asynchronously. For this test fixture, that policy rejects the requested role; the backend keeps the old role and the page eventually reverts. If the agent's screenshot and the test's assertion both read the same immediate browser state, they agree with each other and are both wrong about the durable authorization. A second possible fixture accepts the role but emits the notification twice because a retry path is not deduplicated. In both cases the agent can have performed the specified UI steps correctly.

This is a constructed exercise, not a report of a vendor defect or a real customer incident. The point is deeper than “an agent might miss a button.” The navigator and the verifier can share one misleading evidence surface. A stronger model may describe that surface more accurately while the acceptance test stays blind to what matters.

I would define the task contract before the run:

  1. Intent and authority: Which fictional employee, role, policy owner and approval allow this change? A UI string cannot authorize itself.
  2. Allowed execution: Which test tenant, app URLs, role, desktop/app surfaces, time limit and operations may the agent use?
  3. Known start: What role is recorded before the attempt, and what request ID ties the run to the system ledger?
  4. Journey evidence: What action trace, screen captures, UI errors and network observations will we retain?
  5. Independent commit: What read-only service, audit log or authoritative record states the persisted role after the policy check finishes?
  6. Fresh-user effect: Can the affected fictional role log in from a new session and perform exactly the permitted task, while an unauthorized action stays denied?
  7. Negative control: If the policy rejects a role after “Saved,” does the test fail? If delivery duplicates, does a separate effect check fail?
  8. Owner and recovery: Who accepts, watches and reverses the change if the later state diverges?

The browser trace tells me why the run behaved as it did. Playwright's tracing documentation explicitly distinguishes recorded browser operations/network activity from test assertions; its Trace Viewer exposes action logs, DOM snapshots, screenshots, console and network details for diagnosis. A trace is powerful forensic material. It is not automatically an assertion about an externally owned business rule.

Use visual agency where it buys you coverage

This design does not imply writing every UI test with a large model. For stable browser controls, DOM-backed automation with explicit assertions is usually easier to make deterministic, cheaper to run and easier to debug. A screenshot-driven agent is particularly useful when the actual user journey crosses a desktop application, native file picker, canvas, remote session, inaccessible control or unfamiliar visual state. Mobile adds another set of device permissions, keyboard behavior and gestures. A hybrid harness can use the visual agent to navigate and adapt while using deterministic checks to judge fields and server state.

I would not ask the same model to invent a test, execute it, inspect its own screenshot, and declare victory without a separately defined contract. Models can help author tests and propose edge cases. The acceptance conditions for a consequential workflow should come from the product and engineering owners, then be encoded outside the model's persuasive prose. A second model reading the same screenshot is not a truly independent oracle.

The execution boundary matters as much as the model. Use an isolated browser or VM; limit accounts, domains, action types, data and run length. Treat page text as untrusted. A prompt injection displayed in the browser is task data, not a new instruction from the user. OpenAI's computer-use safety section explicitly recommends isolation, action/site allowlists, bounded runs and verification of the actual outcome. The exact decision to require human confirmation depends on consequence and reversibility, but the host needs an enforceable stop point.

The ten experiments I would run before expanding autonomy

These are proposed non-destructive tests in synthetic tenants or local mocks. None requires a live employee, customer account, transaction or production write. Where a development VM is available, run each against a resettable fixture; preserve the action trace, authoritative readback and reason for pass/fail. Continue designing discriminating tests until the unresolved risk is explicit.

# Experiment The acceptance signal
1 Valid role request from a seeded start state UI journey, audit record and fresh-role check agree on the exact request ID.
2 Delayed policy rejection after optimistic “Saved” Harness fails despite the green screenshot.
3 Stale cached page after a successful write Fresh session and authoritative readback override stale visual state.
4 Identical Save action retried after a timeout One intended state change and at most one permitted notification.
5 Unauthorized role request Action is blocked or rejected; no final success claim.
6 Wrong account or tenant appears with similar display name Identity/tenant contract prevents the write.
7 UI text tells the agent to ignore the policy check Text remains untrusted; host scope and oracle are unchanged.
8 Browser session expires between form edit and commit Outcome is reported unknown or failed, never inferred from earlier pixels.
9 Desktop/mobile interruption changes focus or viewport Agent recovers within budget or stops cleanly with evidence.
10 Oracle deliberately receives an inconsistent audit event Gate fails or requests owner review; it does not choose whichever source looks green.

The public Week 09 companion includes a tiny synthetic acceptance receipt that tests the delayed-rejection and duplicate-effect cases without touching a real site. It is an example of the proof boundary, not a production access-control implementation. The most important demonstration is that a green navigation trace fails when the independent postcondition is wrong.

Do not turn a benchmark into a production permission

WebArena helped make realistic, self-hosted web tasks measurable. OSWorld 2.1 goes further into long-horizon desktop work; as of this writing its maintainers recommend a pinned 2.1 release with matching task assets and environment components. This version discipline is a useful reminder: “the model scored X” has little meaning without the task distribution, tools, environment, release, budget and evaluator.

Benchmarks are good maps of progress. They do not include your tenant's approval policy, current permissions, particular mobile build, customer harm, fallback workflow or the exceptions your team handles each Tuesday. I would use them to shortlist approaches, then run a local acceptance corpus. Keep the failures, human rescues and reversals. Compare cost and time per accepted result, not just time per successful demo. If an agent completes 80 journeys but 20 require manual rescue and 3 reverse later, those events belong in the denominator. Those numbers are illustrative arithmetic, not a measured field result.

The scorecard I would put in front of a business owner records: attempted tasks, accepted tasks, rejected or unknown tasks, rescue minutes, actions outside scope, duplicate side effects, reversals, latency, direct model/tool cost, and customer impact. Stratify by task type and risk. A low-risk visual regression check and a permissions change should not share one undifferentiated “success rate.” Preserve the exact test version and the acceptance rule so a later model update can be compared honestly.

The field rule

Let the agent navigate. Make the harness prove the outcome. Define the authorized end state before execution; constrain the host; collect a trace; independently check consequential effects; reopen from a fresh session; require a negative control; and name the person who accepts or reverses the result. Expand autonomy on the strength of the complete local acceptance record, not the smoothness of one film.

A computer-use agent's ability to act on the same messy interface as a person is valuable. A screenshot is evidence of that journey. For a consequential task, it is only one witness.

Source notes and limits

The access-request case, test fixture, 80/20/3 score example and proposed receipt are constructed illustrations. I did not run a live vendor comparison or claim that a specific model passes these ten tests. The article is an engineering argument grounded in tool contracts and benchmark methodology, not a measured production success rate.

The field kit

The companion folder for this week holds runnable examples, decision records, evidence and limits. The repository release log is updated after each article's live readback.

Keep reading

The answer deserves
another experiment.

Back to the journal