Measure the split before you redesign the agent loop

Watch for a slow agent loop with an obvious culprit. When every computer-use step takes five seconds, the server looks guilty, and a push channel looks like the cure. Before anyone builds that, split the five seconds. In the case below, the server and the desktop agent together accounted for about a third of a second per click. A push channel would have saved under a tenth of a second.

The pattern

An agent was running a QA check on a desktop through GoodQA's computer-use path. Each step went the same way: lease the machine, click a button, take a screenshot, copy the image back, look at it, click again. Every step took about five seconds.

Three real actions on the machine took 13 tool calls and 144 seconds. That is roughly 11 seconds per call: 3 to 7 seconds of tool time, plus 4 to 10 seconds of the model reading a 4K screenshot and deciding what to do next.

The obvious guess was the server. The agent long-polls for commands, so a push channel (websockets or server push) felt like the fix. It would have been a real project, with a new protocol, reconnect logic and new failure modes to review. And it would have been the wrong fix.

Why it happens

A five-second number is a single clock. It tells you the loop is slow, not where. The part of the system you designed most recently, or understand least, attracts the suspicion. Meanwhile the time hides in the glue: credential lookups, connection setup, process startup, image transfer, and the number of turns the model takes.

The fix for that bias is two clocks. Time each call end to end, and read the server's own timestamps for the part inside the server and agent. Whatever is left over is the helper path you wrapped around them.

How to detect it: time each verb two ways

Before anyone designed anything, the owner agent built a small timing harness. One script split each HTTP call and read the server's own created, sent and completed timestamps. Another drove the real multi-hop path used that day, through an ssh hop and the GoodQA CLI to the GoodQA server and the desktop agent on a QA VM. Each verb was timed separately.

The raw CLI run agrees: clicks of 5,361, 5,490 and 5,044 ms, and screenshot plus copy of 6,996 to 8,020 ms. So 85 to 95% of every mechanical call was overhead outside the server. The click itself took about a third of a second.

Where the other 4.7 seconds went

The plan ranked the bottlenecks by what was measured, not by what anyone guessed:

1. A credential fetch on every call. Each CLI call read its secret from a cloud secret store: 0.94 to 1.71 s. One identity needed extra lookups plus a login call, so 3.1 s.
2. Turn count. Act, screenshot, copy the image and read it back were separate calls, and each turn added 4 to 10 s of model time.
3. A fresh ssh connection per call: 0.77 to 0.86 s. A reused, multiplexed connection measured 0.11 s. Every image also got its own file copy, about a second more.
4. 4K PNG screenshots. These were 1.54 to 1.70 MB, and downloading one took 0.85 to 4.1 s. In a bench of the agent's own encoder on real 4K screenshots, full PNG took about 310 ms. JPEG at quality 75 and 1280 px wide took about 47 ms and came out at 105 to 120 KB.
5. Artifact reads had a fixed 0.4 to 0.7 s of server latency, even for a 10 KB image.
6. CLI startup took 0.31 to 0.41 s, mostly imports.
7. An extra round trip. A create request followed by a separate wait request added a round trip, even though the server already accepted a wait time on the create. Each call also opened a fresh TLS connection (about 60 ms).

Item 8 is the one that changed the plan. Not a bottleneck: server queue delivery. From created to sent, the median was 33 to 100 ms. The agent already long-polled, and the server already woke it through database notifications. A push channel would have saved under 0.1 s, so it was rejected.

The push channel died because the plan had "33 to 100 ms queue, under 0.1 s to gain" written next to it, not because someone argued well. That is the value of putting "not a bottleneck" rows in a plan, with the number that rules each one out.

The second surprise was that the agent and server already supported most of what was needed: JPEG screenshots with quality, maximum width and region; click by element; filtered UI-tree queries; wait-for conditions; and a wait time on command creation. JPEG with maximum width was verified live on the agent version in use. The first phase needed no change to the desktop agent at all. The slowness lived in the helper path.

How to prevent it: fix the glue, then the turns

The design put a session daemon next to the CLI, plus a thin client:

• Token in memory. It is fetched once through the existing login path, refetched once on an authorization failure, and then the daemon stops. It is never written to disk, logged, put on the command line, or returned in a reply. Replies pass through a redactor.
• One keep-alive HTTPS connection per daemon. Commands are created with a 28-second wait, so one round trip usually completes them. Command-creating requests retry at most once, and only with the same client request id.
• A local socket in a private directory. The daemon refuses to start if the directory is not owned by its user or is open to group or others. The socket is readable and writable only by its owner. Every accepted connection checks the peer's user id against the daemon's own, and on a mismatch closes with no reply. A lock file keeps one daemon per identity, and a stale socket is removed only while that lock is held. The daemon exits after 15 idle minutes and has a hard limit of 8 hours.
• A standard-library-only client that runs over a persistent, multiplexed ssh connection. Image bytes stream back on the same channel, so there is no separate copy.
• A combined act verb that runs up to 20 steps under one lease, stops at the first failure, and can end with one screenshot. This attacks turn count, not just transport.
• JPEG by default at the display's point width, so image pixels match click coordinates. PNG is an opt-out.

Leases and grants stay enforced by the server. The daemon still refuses input verbs without a lease, and it still journals before each device effect. For more on keeping a credential away from the agent itself, see let an agent ask for a credential without ever seeing it.

Prove the guards, not just the speed

A daemon that holds a privileged token and listens on a socket can quietly become a local privilege problem. So each security claim got a mutation test: break one guard on purpose and confirm a test fails.

In the second mutant run, 22 of 23 mutants were caught, including a disabled peer check, a world-writable socket, a disabled directory check, a removed lease guard, a skipped journal entry and a PNG default. One survived: a daemon in the middle of shutting down kept accepting work, and no test noticed. The next run added tests, and all 25 mutants were caught, including that one and two new ones. The fourth run caught all 25 again.

A separate live security check on the host confirmed: the run directory private to its owner, the socket private to its owner, a connection from another user refused with a permission error, the process environment not readable, core dumps disabled, and no token-shaped strings in the command line or the 18 KB daemon log.

The result, and what it does not prove yet

On the same QA VM, with the same kind of workload, ten back-to-back clicks measured 465 to 524 ms. Earlier clicks in the same run took 525 to 646 ms. UI tree took 540 to 771 ms. Screenshot to a local file took 1.6 to 1.9 s, down from 7 to 8. Click plus screenshot in one call took about 2.0 s, and release took 0.4 s. Lease was still 3.2 s, and the cold-start ping 1.5 s. The task ledger records clicks at 0.4 to 0.9 s, down from 5 s, on the QA VM.

Two caveats belong with those numbers. First, the ledger marks this ready, not shipped: the work was ready for review, and not every supported platform had a recorded live check. Second, the plan's 4K numbers came from existing screenshots and an offline encode bench. Uploads from a remote desktop over a slower link were not measured.

Turns are a latency metric

Model time per call was 4 to 10 s, which is larger than every transport overhead. So the number of turns matters as much as the speed of each turn. A faster click helps. A verb that does click plus screenshot in one turn helps more. When I judge what a computer-use agent can verify per step, turn count is part of the answer, a point I made from another angle in what computer-use agents can prove.

Checklist

Before I redesign an agent loop, I check:

• [ ] Each verb is timed end to end and inside the server, and both numbers are in the plan.
• [ ] The plan lists "not a bottleneck" items, each with the number that rules it out.
• [ ] Someone has checked what the server and agent already support before new protocol is written.
• [ ] Credentials are cached in a long-lived local process, never on disk, in logs or on the command line.
• [ ] That process sits behind a peer user check and a private directory, and each guard has a mutation test.
• [ ] Connections are reused: one keep-alive HTTPS connection and a multiplexed ssh channel.
• [ ] Screenshots default to a compressed format sized to the click coordinate space.
• [ ] Turns per task are tracked next to milliseconds per call.
• [ ] Nothing is called shipped until the slowest supported platform has a recorded live check.

Go deeper: timing agent loops and hardening the helpers around them is part of the Transition2 course.