Landed is not done: define done before agents report it
Watch for completion words that only the author can produce. Landed, merged, tested, briefed, proven: each of them can be true while the thing still does not work on the path a user actually uses. When a status file is built from those words, it fills up with finished-sounding items, and people read them as outcomes.
I opened a fresh ledger for one purpose: take every item the fleet had marked finished and drive it to a state I could believe. Before writing a single task row, I copied the project's status file into it and marked each claim with what it actually was.
Seven rows came back as false claims.
- A key part of a bug-briefing service was marked "proven live" while 217 of its 246 runs had failed.
- A fleet metric said "thirteen active." It was one live session and twelve stale rows.
- A hygiene gate had "landed," but nothing called it.
- A fix to a QA bot was marked "fixed." The bot's own test still failed that check on the commit in question.
- A plugin sign-in feature had a "proof click-test" that turned out to be an offline model of the flow, not the flow.
- A feature that turns system errors into bug tickets was marked done with no live check of the feed.
- 33 tickets recording my own instructions had been bulk-set to a production status with identical millisecond timestamps.
None of these were lies in the usual sense. Each was a completion signal that someone, human or agent, read as an outcome.
Why it happens
The vocabulary available to an agent for describing progress was all author-side. None of the words required anyone other than the author to see the thing work where a user would see it. I had a working rule for how the fleet should operate: checkable goals, a named owner, tested on the exact version that ships. The goals were checkable in principle, but nobody had said what "checked" meant, so "done" had no checkable meaning.
Three earlier attempts each helped and each fell short.
Visibility. I paused the fleet and wrote a restart plan. Every session got a pause guard on every tool call. Each work item got exactly one owner. Each restart step was gated on one overview screen coming back clean: fleet, tasks, live agents, host load, pauses and real versus notional usage. Visibility helped. It did not define done.
A rule scoped too narrowly. A run of release waves kept shipping partial fixes. One passed its local build and then failed when building the deployable image. The rule that came out of it was good: a release fix is not fixed until the deploy's real build and smoke tests pass on the exact commit, and the class of failure is closed, not just the instance. But it applied to release fixes only. Everything else kept the old meaning of done.
Capacity triage. During a catch-up push, my strongest reviewer model was unavailable for several days, so an independent Sonnet reviewer stood in, and implementation and review moved to a Codex pool when Claude sessions hit their usage limits. The plan added a finisher rule: each owner actively drives about two delivery items at most, and nothing new enters a slot until the current item has a deployed version, a live flow and a receipt, or a concrete blocker. That was the right idea. Three days later the ledger showed it was not enforced. The status file still trusted words like "landed" and "proven."
How to detect it
Two smaller episodes taught me more than the big ledger did.
A check that is too strict throws evidence away. An agent was pairing a machine into GoodQA remote control. Its final report was candid, which is the only reason the details are known. The pairing request returned HTTP 201. The agent's one-off handler expected 200, treated the response as a failure, and discarded it before saving anything. A retry for the same handle returned 409, which the agent inferred from source code to be a duplicate-request rejection, and it said clearly that this was an inference, not a captured error body. A replacement request then sat out its ten-minute expiry with eight polls returning pending. No key was saved. The agent committed a helper with seven tests. A check that is too strict loses evidence just as surely as one that is too loose.
Drift shows up only when someone asks. In one session I had asked for three things: the GoodQA bugs, full remote control and full browser support. The session put its time into packaging and harness work. When I asked directly what it was doing, the reply was the most useful sentence of the day: the GoodQA fixes were tested, but they were not live yet, and full browser and remote-control behavior was not fully proven. The status shown up to that point had not separated tested from live.
From those and the ledger, these are the signals I now audit for:
| Signal | What it usually means |
|---|---|
| "Proven live" next to a high failure count | The status was written once and never re-read against the system of record |
| A count of active things that never drops | It counts rows, not live sessions |
| A gate that "landed" | Nobody checked that anything calls it |
| A bulk status change with identical timestamps | A script set the status, not a check |
| A "click-test" that runs offline | A model of the flow, not the flow |
| Tested, with no mention of live | The author's view only |
How to prevent it
The fix was a definition, applied everywhere at once.
One sentence for done, in every session. I added a tenth core rule, rendered into every runner session's context: never report done unless the work is click-tested live from the user's side. For anything without a UI, such as a gate, guard, verifier or background unit, it is done only when its negative control is observed firing on the installed path, not when a unit test passes. A dark or flag-off feature is "landed-dark," and that is not done. Agents are told to say "landed, not yet click-tested" when that is the truth. The completion ledger adds the piece I care about most: the negative control has to be observed by someone other than the author. "Landed" and "briefed" are never done. A related argument about where proof belongs is in a merge gate must prove the tests ran.
Production-only proof moves out of the coding task. The new rule created a new failure almost immediately. A click-test repair task had production-only steps in its brief's done-when section: a live timer run, a drill and a real stage. A coding session has no production access, so every review failed on a step the author could not perform. The rescue agent put it plainly: as long as those steps stayed in done-when, every review would fail. The production steps moved to a separate coordinator step that runs after the code lands. The offline part kept its own gates, and both suites passed (44 of 44 driver tests, 124 of 124 post-ship tests). The live tree already re-runs a failing click-test group after a driver change, so landing the fix triggers the live run on its own.
This is the extension I did not teach before. Spec-driven coding asks for an outcome, acceptance criteria, negative cases, evidence and sign-off. A criterion that only production access can satisfy does not belong in a coding task's done-when. Put it in a step that follows the merge. Otherwise the review fails forever on something the author cannot do, and people learn to ignore red reviews.
Honest intermediate words. Runner reports now end in a form I can trust. One Sonnet-authored click-proof task closed with: "LANDED-READY, not DONE. Nothing here has run against the live site." It then named the owner who would close it, the post-ship command they would run on the live version, and the bar: all 5 flows pass. It names what is missing, who closes it and how. Several other tasks from the same week end with "landed, not yet click-tested."
Turn assertions into live flows. One product decision was that a set of commercial features stays switched off until a later launch, and that switching them on must be a single click-tested configuration change. "New workspaces start with those features off" had been an assertion. A runner task turned it into a post-ship flow that checks four things: a new workspace's entitlement record shows the off state with no leftover keys from the on path; the settings page shows none of the gated controls; one public route answers with a 307 redirect to its section of the landing page; and the landing page says the feature is not open yet. The task shipped with negative-control commits and mutation evidence. That gave the off state a receipt, and it gave the later switch a baseline to compare against. The task itself was at "implementation complete at a reviewed commit," which is not done, and I have not confirmed that the flow has since run green live.
Why this works: it measures outcomes with independent evidence instead of completion signals, it grades by end state in the system of record, and it makes the top evidence level (a live user-flow observation) mandatory rather than optional. For what a browser-driving agent can and cannot establish, see what computer-use agents can prove.
Checklist
- Define done in one sentence before any agent starts, and render it into every session. Mine: click-tested live from the user's side, or a negative control seen firing on the installed path by someone other than the author.
- Give agents honest intermediate words (LANDED-READY, landed-dark, "landed, not yet click-tested") and require each to name what is missing and who closes it.
- Audit status claims against the system of record on a schedule. Look for metrics that count rows instead of live things, and bulk status changes with identical timestamps.
- Split production-only proof out of coding briefs into a coordinator step that runs after landing.
- Treat "what are you doing?" from a human as a missed checkpoint. The agent should have said "tested but not live" first.
- Check status codes by meaning, not by exact value, and never discard a response before saving it.
- Turn each standing assertion about product state into a live flow with a negative control.
Go deeper on definitions of done that agents can actually meet in the Transition2 course.