Eric MacDougall Follow the work
All field notesFIELD NOTE / 014

Engineering ownership / Days 89–95

The demo worked. The engineering problem began afterward.

Why a convincing agent-built demo is the beginning of product ownership and engineering evidence.

Eric MacDougall14 minute read · 2026-09-25
EVIDENCE / DECISION
An agent's work passes through independent controls before a release decision
FIELD NOTE 14 / ENGINEERING OWNERSHIP
THE OVERVIEW FILMWatch the full argument

Approved overview film · Open the direct MP4 ↗

Read the complete film transcript

The demo worked. The engineering problem began afterward — complete spoken transcript

Published transcript of Eric's approved overview film. Non-spoken pause cues are omitted; the exact audio is available in the player above.

The approval screen says active. The access should be revoked. Picture a fictional small company replacing a maintained approval service with an internal tool built by coding agents. The demo works. The generated tests pass. A second model run agrees on the migration. But an old record says approved true because it was approved once, before a later revocation. The new interface converts that historic flag into current access. Nothing has crashed. The software is consistent with its incomplete brief. The company now owns a product it cannot explain from the screen. This is the hidden cost of calling a plausible first build finished.

Agreement between two strong models can sharpen a hypothesis. It cannot recover a policy that never entered either context. For this migration I would hold out four historical cases: a valid approval, an approval later revoked, a legacy true value with no provenance, and a record with no approver. A real policy owner labels the permitted new state and the evidence required for each case. Ambiguous records stay unknown; they do not become active because a green screen needs a value. Keep that acceptance set outside the worker's editable fixtures. If a deliberately bad migration grants access on the revoked case, the independent test must fail. That is the difference between a demo and release evidence.

Six months later a customer reports, I was approved but I cannot get in. A new agent can propose three convincing repairs: rerun a notification, set the entitlement manually, or change the policy evaluator. A codebase alone does not tell an operator which is right. Trace the request, the policy version, the entitlement command and the authoritative readback. Was the decision wrong, was its effect lost, or did the user interface misreport a correct state? Only after locating that boundary should the team prompt a fix. Otherwise every incident becomes a fresh archaeology project, and every apparently small repair can create a second inconsistency.

Saving the conversation does not solve continuity. A note that says customer approved may outlive the policy version and the later revocation. Even a durable summary can be durably wrong. The operating record needs source identity, effective time, exception, owner and supersession. Keep design rationale, action journal and customer-state truth separate. On resume, give the agent a narrow packet describing what was intended and which facts need revalidation. Then query the current protected record at the action boundary. A markdown memory file can help a worker remember the story. It should not become the source of truth for a customer's entitlement.

The first version is the easiest one to celebrate. The second change tells you what was built. Suppose the business adds a temporary approval mode. In the first design, one versioned decision service owns the new transition and its contract. In the second, copies of the same rule live in a web controller, a nightly job, an export and a notification worker. Agents can edit either system. The difference is how much re-prompting, tracing, retesting, redeploying and adjacent-fault hunting the team must do to accept that change. Measure affected contracts, review cycles and deployed readbacks. Deliberately break the revoked case in a test. If the gate stays green, the tests are decoration. Fast first code is not the same as a cheap second change.

This gets worse when everyone with a coding agent starts replacing every small SaaS subscription. The salesperson builds a scheduler instead of selling. Operations builds a reporting portal. Someone else builds a miniature CRM. The first drafts are cheap; each becomes a product with access control, backups, vendor integrations, incident support and a future migration. I would put the subscription bill beside owner hours, the cost of interrupted core work, security and recovery duties, and the exit path. A maintained tool does not eliminate business policy ownership. It can remove commodity implementation and maintenance work. Build custom when the workflow truly differentiates and you can fund a named owner. Choose the maintained tool when that full comparison favors it.

An agent can make the approval endpoint faster while the customer still waits on a manual exception queue. The benchmark improves; the outcome does not. That is what happens when the builder cannot explain the product's causal path. Measure request age, queue age, exception types, owner intervention, entitlement readback and customer completion together. Find the controlling bottleneck before optimizing. The team should also measure the time to repair a failed state and the effort of the next change. DORA's research describes AI as amplifying the surrounding organization; it does not calculate this fictional company's return. The operating system around the model is part of the product.

After the demo, ask five questions. Is the workflow differentiating? Do owner-labeled historical cases and current readbacks support its behavior? Can a new maintainer make the next change without rediscovering every rule? Who handles incidents and policy changes? How will the company exit or retire the product? Build when the advantage and evidence justify durable ownership. Buy when a maintained commodity meets the need at lower total owner cost. Stop when neither option has a credible owner for policy and operation. Here is the rule: use agents to accelerate implementation, but judge the product by the next change and the next incident. A working demo becomes engineering only when someone can explain it, test it, operate it and leave it safely.

The demo worked. The engineering problem began afterward.

The new internal approval tool looks finished. A customer requests access, the screen turns green, and the generated tests pass. Two independent coding runs agree on the implementation. The team has a persuasive demo and a growing temptation to cancel the SaaS subscription.

Now open an old account. Its approval flag is true, but the approval was later revoked. The new tool treats the flag as a current entitlement. It did exactly what the brief and its generated fixtures implied. The brief omitted the part of the business that mattered.

This is a synthetic case, not a reported customer incident or a measured failure of a named model. I use it because the hard engineering question starts when the first version appears to work: who can explain, change, operate, and eventually retire the system? A model can write a great deal of code. Agreement between model runs can surface disagreements and sharpen a design. Neither can recover an unstated historical rule or take ownership of a service after the original conversation is gone.

The surface can be right while the product is wrong

Imagine a small company replaces a maintained approval and scheduling service. The old systems have several kinds of evidence: a current approval, a later revocation, an ambiguous legacy Boolean, a missing approver, and a customer-specific exception. A screen that says Active compresses those histories into one word. The operator needs to know which event made it active, under which policy, at what effective time, and whether a later event cancelled it.

I would make the migration contract explicit before asking an agent to implement the UI:

Historical record Permitted new state Evidence required
Valid approval, no later revocation Active, subject to current policy Approver identity, policy version, effective time
Approval followed by revocation Inactive Ordered event history and revocation scope
Legacy approved=true with no provenance Unknown, held for review Owner-labeled migration decision
Missing approver or policy version Unknown, held for review Authoritative source or named owner
Customer-specific exception Whatever the governing exception says Current exception record, scope and expiry

Those rows are proposed rules for this fixture. A real product owner must label its actual policy; the table cannot be copied into a real business and called correct. The useful technique is to hold out records that the build prompt did not mention. If a candidate implementation grants access from an ambiguous flag, a protected acceptance check must fail even if the generated UI and its self-written tests are green. GitHub's agent application card explicitly warns that generated code may look valid while failing the intended semantics. That documentation supports the need for independent review and testing; it does not measure this fictional company.

An engineer needs a causal model, not only a codebase

Suppose a customer has an approval badge but no access. A new agent session offers three plausible fixes: rerun the notification job, set the entitlement manually, or repair the policy evaluator. A person who cannot trace the path from request to decision to effect cannot choose responsibly. A successful edit to a plausible file may hide the real failure, or create a second one.

The operating record should let a new maintainer answer four questions in order:

  1. Intent: What was requested, by whom, under which policy and customer scope?
  2. Decision: Which rule evaluated the request, with which source data and version?
  3. Effect: Which entitlement, notification, export or schedule change was attempted and accepted?
  4. Readback: What does the authority for each consequential state say now?

These are different records with different owners. A screenshot is not an entitlement readback. A model's explanation is not a durable event log. An API receipt does not prove that the customer can use the product. If an effect is uncertain, keep it UNKNOWN and route it to a named owner rather than generating another action from a guessed state.

This is also why “we saved the agent's memory” is an incomplete answer. A note saying “customer approved” needs source identity, version, effective time, exception, expiry and supersession. When a later revocation arrives, the system must invalidate or revalidate the old note. Keep design rationale, action journal and customer-state truth separate. A handoff summary can explain what the last operator believed; it cannot replace a current query of protected records.

The second change reveals the architecture

The first generation was fast. The business now adds a temporary approval mode. In one implementation, the rule lives behind a coherent decision service with a versioned contract. In another, copies of the same conditional appear in the web controller, a nightly job, CSV export and notification worker. Both could have passed the first demo. The second change reveals the difference.

The cost is rarely “a developer hand-recodes everything.” In an agentic team it is repeated prompting, tracing affected paths, rerunning tests, checking the deployed effect, and asking what adjacent behavior was missed. Measure the accepted second change: affected contracts, file spread, reviewer cycles, regression failures, production readbacks and owner time. Do not invent a ratio from a synthetic story. Maintainability sensors for coding agents discusses ways to notice change spread and other maintainability signals; those are engineering indicators to test locally, not universal proof that a particular agent harmed this codebase.

The useful harness has an independent failure path. Give the worker a task and a worktree, but keep owner-labeled approval cases and policy invariants outside its editable working view. Run the same cases after a second change. Mutate the code deliberately so a revoked approval grants access; the suite must go red. Re-run the deployed readback for the actual customer outcome. If tests remain green after the mutation, the team has a vacuous gate, however impressive the demo looked.

The real bill includes diverted attention

Giving every employee a coding agent can turn every ordinary SaaS friction point into a small internal product. A salesperson starts a scheduler. Operations builds a reporting portal. Someone else creates a miniature CRM. The first drafts are cheap, so the company accumulates tools that each need access control, backups, support, integrations, migration and an owner when the original builder leaves.

I would compare two complete options for the same business outcome:

Cost or capability Maintained tool Internal build
License or hosting Subscription and usage Compute, storage, service dependencies
Implementation Configuration and integration Design, acceptance rules, implementation
Ongoing changes Vendor roadmap and integration work Owner time, tests, deploys, incidents
Security and recovery Vendor commitments to verify Team-operated access, backup, restore, response
Exit Export and migration effort Data portability and shutdown plan
Opportunity cost Time spent adapting the tool Time diverted from the employee's actual role

A license is not automatically cheaper, and internal code is not automatically waste. A custom build can be right when the workflow is differentiating, the vendor cannot meet a material need, and the organization can fund durable ownership. The trap is comparing the monthly invoice with only the first AI-generated draft. Include the salesperson's missed selling time and the engineer's future investigation time. Use observed owner hours and actual vendor terms; this fixture supplies no price claim.

The 2025 DORA research describes AI as amplifying existing organizational strengths and weaknesses. It is a reason to examine the team's review, feedback and operating system, not a number that can be applied to this case. Code generation may accelerate a good engineering process. It can also accelerate the creation of half-owned commitments.

Optimize the customer path, not the attractive metric

Six months later someone asks the agent to speed up approvals. It reduces endpoint latency in a synthetic benchmark. But the customer waits on a manual exception queue fed by stale records. The endpoint is faster; the outcome is unchanged.

Before choosing an optimization, trace request latency, queue age, exception classes, owner interventions, failed notifications and the final customer result. Ask which boundary controls the end-to-end time. Require a human-readable causal account of the state machine. A model can propose options and implement the chosen change, but the owner must define the outcome and the independent readback that will tell the team whether it improved.

The same rule applies to expansion. If the team cannot explain what makes an approval valid or how to repair a failed entitlement, it does not yet know how to safely add another customer cohort. Scale multiplies unresolved semantics.

Make the build, buy or stop decision explicit

After the demo, I would ask for a one-page decision with five sections:

  1. Differentiation: Which capability is specific enough to this company to justify ownership?
  2. Correctness evidence: What owner-labeled historical cases and deployed readbacks were passed? What remains unknown?
  3. Changeability: Can a new maintainer add the next mode without rediscovering rules across the codebase?
  4. Operations: Who handles failures, policy changes, access, backup, restore and customer communication?
  5. Exit: How will data be exported or migrated, and when will the team reconsider the decision?

There are three defensible outcomes. Build when differentiation and evidence justify a named team. Buy when the maintained commodity solves the problem at lower total owner cost. Stop when neither option earns the business outcome yet. Unknown records or costs should remain visible as UNKNOWN; a polished model debate must not quietly turn them into a green light.

This is the steadfast lesson: use agents to accelerate implementation, then judge the product by the next change and the next incident. Ship the internal system only when an accountable owner can explain its state, test its historical edge cases, operate its recovery path and exit it. Otherwise the impressive demo is a prototype, not an engineering result.

A bounded experiment before a stronger claim

The Week 14 MIT companion contains the synthetic approval-workflow fixture and its readbacks. The fixture covers owner-labeled history, policy versions, context handoff, a deliberately defective migration and an owner-time worksheet; no live customer system is implied.

I would run ten probes: a valid historical approval; a later revocation; an ambiguous legacy flag; a missing policy version; a changed requirement after the demo; resume from a thin prose summary; resume from a versioned decision record; a stale handoff; a second-change effort comparison; and a full owner-time comparison against a maintained tool. For each, log the invariant, input, expected state, observed state, intervention and operator time. A passing local fixture proves only that the fixture's code handled those cases. A claim about a real product would need an authorized staging target and its own evidence.

The NIST AI Risk Management Framework calls for defined human roles, monitoring and regular evaluation across the system lifecycle. My practical version is simpler: name the owner, preserve the decision record, and keep the independent test and recovery path alive after the original model session has disappeared.

The field kit

The companion folder for this week holds runnable examples, decision records, evidence and limits. The repository release log is updated after each article's live readback.

Keep reading

The answer deserves
another experiment.

Back to the journal