The promise of agentic coding is real. A capable worker can inspect a repository, implement a change, run checks, and return a patch in the time it used to take a team to finish the first meeting. That changes what an engineer can attempt in a day.
It also changes what can go wrong in a day. The patch, the tests, the summary, and the apparent evidence can all come from the same run. A green transcript can feel like four independent confirmations when it is really one assumption repeated four ways.
After more than twenty years across engineering and technical leadership, the question I keep coming back to is not whether an agent wrote code. It is who owns the test that lets the code ship. This is the operating pattern I would use with a strong agent, a lighter model, or a junior developer moving quickly with an agent at their side.
First, stop blaming the model for ordinary engineering mistakes
Two buyers reaching for the last unit of stock is not a deep agentic failure. A properly guarded database write can let one allocation succeed and make the other return no row. Under PostgreSQL's default Read Committed isolation, a competing UPDATE can wait, then re-evaluate its condition against the updated row. A conditional allocation such as UPDATE inventory SET available = available - 1 WHERE sku = ? AND available > 0 RETURNING sku is a normal engineering pattern. PostgreSQL's transaction behavior and UPDATE reference establish the mechanism.
The same goes for payment retries. A lost response after a successful charge is a known distributed-systems problem. A stable operation identity and provider reconciliation are established ways to handle it. Stripe's idempotency documentation and Amazon's retry design discussion are useful references.
So why discuss these cases in an agent workflow? Because a team can teach the pattern to every new engineer—or encode it once in the gateway, database permissions, and acceptance contract so every implementation has to satisfy it. A lighter reasoning run can miss an edge. A junior developer can miss it too. Neither fact makes the edge uniquely “AI.” The agent-specific risk is that a rapid run can generate the implementation and its own favorable evidence, while the reviewer is asked to sign off at machine speed.
Give the agent a task. Keep the verdict elsewhere.
Suppose the acceptance condition is: one checkout intent yields at most one provider charge, even if the provider accepted the request and the network dropped the acknowledgment. The agent should be free to choose its code structure and add unit tests. The condition itself should be recorded before the implementation and kept out of the agent's writable checkout.
The independent test then injects the dropped acknowledgment, retries with the same persisted intent, reads the provider ledger, and checks that exactly one charge exists. A browser confirmation alone cannot establish that outcome. In a stock flow, a separate contract should exercise concurrent requests and a bypass attempt, then read the authoritative inventory state. This is how a familiar pattern becomes an enforced team standard.
There is a research reason to scrutinize this boundary. A study of agent-generated tests in a particular SWE-bench Verified setup found that those tests largely supplied observational feedback, and adding more test writing through prompts did not significantly improve results in that setting. That is a result about the studied setting, not proof that agent-written tests are useless. Read the study. NIST also documents evaluation loopholes and cases where agents have modified tests or scoring code. That does not mean every coding agent is deceptive; it means the grader needs its own custody. NIST's evaluation-integrity discussion.
My rule is simple: let the worker create useful tests; do not let the worker be the sole author and keeper of its final exam.
Hooks enforce covered calls. They are not a magic perimeter.
Instruction files—AGENTS.md, CLAUDE.md, and skill documents—are useful for communicating architecture and norms. They are still instructions. For an action that must be denied, I want an executable control at the point where that action is attempted.
A pre-tool hook can inspect a proposed operation and refuse a covered call before it executes. A post-tool hook can record what happened and trigger a follow-up check, but it cannot undo a completed side effect. Both Claude Code's hook reference and Codex's hook reference document pre- and post-tool events; their coverage and behavior differ by runtime. Cursor documents its own local, project, and cloud hook surfaces. A local user hook does not automatically travel into a cloud worker. Cursor's hook reference.
That difference matters more than the name of the product. If you tested a deny on your laptop but the deployed worker uses a different route, you have verified the laptop. Make a route table: action, execution surface, hook that should see it, expected deny behavior, and actual observed result. Try a harmless forbidden action on every runtime you intend to trust. Retain the trace. If a route cannot be intercepted, restrict its credential or move the guard into the service it calls.
A concrete implementation can use a short Python pre-tool script to resolve the proposed file path, compare it with the assigned worktree root, and exit with the documented block signal when the path escapes. That does not make arbitrary shell execution safe: shell access may require a separate sandbox or narrower tool permissions. An after-tool script can append a write-ahead receipt with action, run ID, result, and timestamp to storage the worker cannot silently edit. These scripts are small enough to test with a normal path and a deliberate escape path. They are stronger than hoping a Markdown sentence gets read at the right moment.
A worktree is a workspace boundary, not a security boundary
I give each run a separate Git worktree so patches, temporary files, and test results do not step on each other. Git documents the linked-worktree model. The launcher records the assigned commit and worktree root before handing work to the agent. That makes later evidence attributable to one attempt.
But a worktree does not stop a process with broad filesystem rights from reading a sibling checkout. Scope the tool and operating-system permissions too. For consequential tasks, the independent grader and its fixtures live outside the worker's writable tree. An external supervisor can hold the acceptance criteria and send focused corrections or missing context to the worker without shoving a huge memory dump into every prompt. The supervisor's own state must be durable: decision, evidence, unresolved question, next experiment, and the exact artifact being judged.
A passing suite needs to prove it ran
Before a merge, the release gate should check more than a zero exit code. Did the expected suite collect any relevant tests? Is the report from this commit and this worktree? Did the independent provider or database readback match the intended result? Does a deliberately broken version of the targeted behavior make that suite fail?
pytest has a distinct exit code for no tests collected; treating an empty run as green is avoidable. See its exit-code reference. Mutation testing formalizes the negative-control question: if I break the behavior I claim to test, does the test notice? Stryker's documentation explains the approach. You do not need a giant mutation campaign for every patch. Pick a fault tied to the consequential invariant.
I built a small illustrative local contract around stock allocation and a fake payment provider. The normal implementation passes. A deliberate bypass of guarded inventory and a deliberate duplicate payment attempt both make the protected tests fail. It is useful proof that the example gate is non-vacuous; it is not a live PostgreSQL, Stripe, or production-agent benchmark. The reproducible commands and limits are in the harness field experiment, alongside a download of the exact scripts.
Computer-use testing needs a second observer
The same evidence problem appears when an agent drives a browser or desktop. A screenshot can show a success banner. It may not show whether a save persisted, whether the wrong account was used, whether a duplicate operation occurred, or whether a background job failed after the banner appeared.
I would define a task in terms of the state that should exist after the UI interaction. For a document edit, reopen the saved document in a fresh session and compare the actual content. For a purchase-like workflow, read the authoritative provider or order record. For a deployment, query the running service at the expected revision. Keep the actor's trace because it helps diagnose a failure, but let another observation source decide the result.
This is a design recommendation, not a claimed failure rate for a named computer-use model. The point is to prevent the easiest false positive: the same agent that clicked the button announcing that the click worked.
The operating loop I would put in place
- Write the consequential invariant first. Describe the business or user outcome in a versioned acceptance contract, including one failure mode and the expected recovery.
- Give each worker a bounded checkout and identity. Record its worktree, commit, permissions, task, and contract version before it starts.
- Put scripts at the action boundaries. A pre-tool script denies covered prohibited calls; a post-tool script records observed calls. Test both with harmless positive and negative canaries in every actual runtime.
- Keep the grader independent. The worker can add tests, but it cannot silently replace the protected fixtures or change the pass threshold.
- Demand non-vacuous evidence. Check collection count, report freshness, independent outcome, and a selected deliberate fault that the test catches.
- Review the decision, not just the diff. A human owner resolves the architecture and product tradeoffs, records why the evidence is enough, and names the remaining risk.
- Feed failures back into the harness. If a bug is a known pattern, encode its contract once instead of asking every future agent to remember the lesson.
That last step is where the leverage comes from. The best workflow does not require a heroic reviewer to catch the same class of mistake every afternoon. It turns hard-won engineering judgment into a repeatable boundary and still leaves a human responsible for whether the boundary matches the product.
Evidence boundary
The checkout and inventory stories above are explicitly illustrative. The local harness exercises SQLite and a fake provider, not a live payment system or production PostgreSQL. Product hook documentation describes available mechanisms, not measured coverage in Eric's deployed accounts. A later field report should add live route canaries, provider readback, and measured exception rates. Until then, the recommendation is an architecture pattern supported by cited mechanisms and a reproducible local demonstration—not a universal claim about all agents.
The field kit
The companion folder for this week holds runnable examples, decision records, evidence and limits. The repository release log is updated after each article's live readback.