The agent timed out after the change went through

The agent reports a failed move. The customer is already serving traffic from the new GPU pool.

Both observations can be true. A caller can lose its response after a control plane accepts a request. If the agent treats its timeout as proof that the target rejected the request, the recovery run begins from a false world model. It may retry, delegate a second move, or undo an effect that is still unfolding. The costly part is not usually the first timeout. It is the next plausible decision made before anyone has established what actually happened.

This field note uses a fictional inference-service move as an engineering exercise. It is not a report of an incident at a named cloud or a claim about one agent vendor. The difficult property is common to distributed operations: transport observation, workflow history, service receipt and customer outcome are separate facts. A model session can summarize one of them fluently while the others remain unresolved.

The approved overview film and its complete transcript are published with this article. The Week 13 MIT companion contains a synthetic recovery journal, ten fault-injection probes and its release-gate readback. No synthetic result should be mistaken for a live cloud acceptance test.

The fictional timeline

The authorized goal is to move one customer's GPU inference service from pool A to pool B. A placement call uses operation move-214 and a precondition on placement version 41. The service accepts it, records an operation receipt, and starts downstream work. The caller's acknowledgment is lost. The agent sees a timeout.

Meanwhile, the account mapping still shows A, the network route has switched to B, and a serving group is draining. One worker reports the old mapping; another reports the new route. A resumed agent receives a compressed summary: “the move failed.” It proposes a second pool move. That proposal is dangerous even if the original placement API correctly deduplicates an operation key. Deduplication answers a narrow question about duplicate commands. It does not prove that account mapping, route, serving and customer-visible health have converged.

The fix is not to ask a larger model to infer the missing receipt. The operator system needs a durable record and an outcome protocol.

A timeout belongs in an explicit EFFECT_UNKNOWN state until the target can be queried. “Unknown” is an engineering result: it blocks an unjustified second action and creates work for a named owner.

A durable journal that outlives the agent

Before dispatch, the host writes an action envelope outside the model's editable conversation:

{
  "case_id": "gpu-move-214",
  "operation_id": "move-214",
  "principal": "capacity-operator",
  "authorized_target": "pool-B",
  "placement_precondition": 41,
  "policy_version": 7,
  "expected_terminal_state": "pool-B healthy for customer",
  "deadline": "synthetic deadline",
  "owner": "capacity-oncall",
  "status": "INTENT_RECORDED"
}

Dispatch, timeout, acceptance receipt, child effect and readback are appended events. A later agent gets a bounded recovery packet with the original operation ID and current unresolved obligations. It cannot replace the attempt with a tidier prose account. The protected store also keeps the principal, policy version and authorized scope; the next model run does not get to decide those from its own prior message.

In an actual implementation, the journal must be durable, access controlled and consistent with the service's operation identity. The sample JSON alone does not solve an atomic write-versus-dispatch gap. The host must test crash points before and after each write and call. An outbox or workflow engine may be appropriate where the application requires stronger atomicity between intent and dispatch, but the target still needs its own readback. Those design choices depend on the actual service contract.

This is also where prompt files stop being enough. An AGENTS.md or operating note can tell an agent to reconcile. A pre-dispatch host check can refuse a side effect with no journal entry; a post-dispatch path can append the receipt or mark the effect unknown; the service can reject a stale version; an independent reconciler can query the target after the agent disappears. Each control acts at a different boundary. Test every route the agent can take, including delegated tools and resumed runs.

Durable engines help without making the external world atomic

AWS Step Functions redrive documentation says eligible unsuccessful Standard Workflows can resume from an unsuccessful step, retaining results and execution history of successful steps. It documents eligibility, exceptions and state-specific redrive behavior; for example, rerunning a Task state can reset its retry count. That is useful recovery history. It is not a universal exactly-once guarantee for an arbitrary external placement service.

Temporal's child-workflow documentation describes parent and child execution relationships, including parent-close behavior. Its Activities documentation covers the external work that workflows invoke. The distinction matters: a workflow event can record that an activity was scheduled or completed according to its own semantics. It does not by itself certify that every dependent service or customer experience now matches the intended outcome.

OpenTelemetry's messaging span conventions can help link asynchronous producers and consumers. Propagate case ID, operation ID and policy/source version through the delegated work, and link spans where messages cross process boundaries. A trace is a way to reconstruct causality. A transport span labeled successful is not authorization to claim a business operation complete. The messaging conventions themselves are documented as Development status; instrument and inspect the actual stack rather than assuming a universal field layout.

The practical distinction is simple: use workflow history and traces to find which effects to inspect. Use authoritative target readbacks and a convergence contract to decide what state the system is in.

Reconciliation has to cover the fanout

For this fictional move, a release contract could require all of these within a defined deadline:

1. The original placement operation ID resolves to exactly one accepted target and current placement version.
2. The account mapping points to the authorized pool for this customer.
3. The network route points to that same pool and carries a version newer than the precondition.
4. Serving instances in the old pool have drained or have an explicit, approved transitional state.
5. Serving instances in the new pool pass a customer-scoped health probe.
6. Billing or capacity accounting reflects the intended allocation without duplicate entitlement.

These are proposed checks for the illustrative architecture, not universal GPU-cloud requirements. They force the designer to name each consequential branch. If one branch is still in flight, the overall result is pending. If a readback is unavailable, the effect remains unknown. If a branch diverges from the authorized target, the case needs a recovery decision. The agent's final turn or a parent workflow's green status cannot silently collapse those distinctions.

The reconciler also needs a deadline and owner. An unresolved effect cannot live forever in a chat log; it needs an operator queue, a retry or inspection schedule, and a customer-impact view. Report both the count of unknown effects and the time to reach a safe terminal state. Throughput alone will hide this debt.

Compensation is a new decision, not an inverse command

Now add one intervening event: while the first move is being investigated, the customer legitimately scales pool B. A blanket “move back to A” can overwrite the newer capacity decision or cause another outage. The original action's inverse is not necessarily authorized in the current world.

Microsoft's compensating transaction pattern is explicit about the limits: concurrent changes can prevent restoration to the original state; compensation is application specific, can itself fail, needs progress records, and high-impact or ambiguous cases may require human judgment. It also notes that a safe forward alternative may be preferable to canceling prior work. That is the right mental model for an agentic operator. A compensating action needs a new current-state precondition, business owner, customer impact assessment, execution receipt and readback. Some outcomes are irreversible; those should be identified before the agent gets execution scope.

An agent can still be useful in this phase. It can gather the four records, explain the unresolved branches and draft the candidate options. The authorization and state checks should not be delegated to its summary. If the facts do not support one safe automated branch, REQUIRES_OWNER is a correct result.

Ten experiments before widening execution scope

The first pass should use a disposable fixture with a protected event log and deterministic target oracle. These tests can be run without touching customer traffic or live capacity. Each probe should assert both the allowed result and the forbidden second action:

For each test, capture the protected journal, service state, agent-visible packet and forbidden-action assertion. A test that merely checks the agent's explanation is vacuous. Mutate the fixture deliberately: allow a second move on lost reply and confirm the gate goes red. If the test cannot detect that regression, it is not evidence for safe recovery.

After the fixture, repeat only a safe, authorized subset against the actual staging control plane. Verify the real hook coverage, persistence behavior, operation-ID lookup, latency and permissions. A local synthetic pass proves the harness logic in that fixture; it does not establish a production recovery rate.

The operating rule

When a consequential tool call times out, record the effect as unknown. Recover by the original operation identity and current target readback before retrying, compensating or declaring success. Keep the intent and observations in a protected journal that survives agent sessions, make unresolved branches visible to a named owner, and test lost acknowledgments after commit rather than only failures before commit.

That is how an agentic operations system earns more execution scope: by showing it can get back to a known, authorized state when the happy path lies.

Sources and limits

• AWS Step Functions redrive, specific eligible Standard Workflow behavior and exceptions (S214).
• Azure Architecture Center, compensating transaction, recovery under concurrent state and human/business judgment (S215).
• Temporal child workflows and Activities, workflow relationships and external work semantics (S216).
• OpenTelemetry messaging spans, asynchronous trace correlation, with documented Development status (S217).

All service names, IDs, versions and outcome claims in the GPU example are fictional. The ten experiments are proposed local tests, not completed live cloud trials. The vendor sources support the described mechanics within their documentation; they do not prove this proposed harness or compare model performance.

The field kit

The companion folder for this week holds runnable examples, decision records, evidence and limits. The repository release log is updated after each article's live readback.