Unknown is an outcome: designing agent systems around lost replies

I model a provider accepting a request and losing the reply as an unknown outcome. The worker’s timeout cannot tell me whether the remote operation happened.

There are now two different facts: the local call failed to complete, and the remote operation may have happened. A system that records only success or failure has no honest place to put that combination.

Our October gateway and jobs work made outcome uncertainty explicit. The gateway exposes outcome_unknown. The jobs module records pending and unknown effects. The ledger refuses evidence-free release after dispatch. These choices connect execution, recovery and budget accounting around a difficult boundary: losing communication does not establish what another system did.

The moment a retry changes meaning

Before dispatch, a local validation failure can often be retried without duplicating a remote action because no action was sent. After a request may have reached the provider, a new attempt can create a second accepted action.

The failure code alone is insufficient. A timeout, connection reset or process death can happen before or after acceptance. Even a downstream HTTP error may be generated by a proxy while the provider continues. The application needs evidence about the dispatch boundary and the provider's contract.

The useful state model distinguishes reserved, dispatching, succeeded, definitely failed before dispatch and outcome unknown. These describe knowledge about an operation rather than merely how a network function returned.

Execution outcome and settlement status may also differ. The response may have completed while usage accounting remains unresolved. Or the connection may have failed while a provider later reports billable usage. A product that merges both axes into one status tends to make budget and retry decisions from incomplete evidence.

Operation identity survives the failed attempt

An operation is the business request. An attempt is one execution of it. A receipt is provider-side evidence associated with an accepted request. These identities should be related without being interchangeable.

If a retry silently generates a new operation ID, local deduplication may no longer recognize that the caller is repeating work. If two genuinely different requests reuse an ID, the system needs a content binding to detect the conflict.

Our gateway requires an idempotency key for admission. Its backend admits and dispatches a durable attempt before the provider stream is invoked. The existence of that key does not by itself prove provider-native deduplication. The gateway, ledger, workflow owner and provider each need a defined contract for their own boundary.

AWS's Builders' Library discusses client request identifiers, semantic equivalence and late-arriving requests in idempotent APIs. It is useful primary context for this identity distinction: https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/.

For an integration, document key scope, retention, material-input binding and behavior after expiry. Reusing a local key is only safe against remote duplication if the actual path preserves the relevant provider guarantee.

Durable intent narrows the dangerous gap

The jobs worker begins by claiming a finite lease and checking domain authority. Before execution it records effect intent. It then checks current ownership and authority again.

The pending record matters because the process can die at any later point. Recovery should see evidence that an effect may have crossed the boundary. Ordinary claiming excludes pending, successful and unknown effects rather than interpreting an absent completion as permission to send again.

The implementation permits another BeginEffect only after a definite failed-before-dispatch state. That is a stronger condition than “the last attempt returned an error.” It requires the adapter's result to carry meaningful evidence about whether dispatch could have happened.

The following is illustrative adapter logic, not a drop-in implementation:

intent = persist_effect_intent(operation, attempt)
try:
    receipt = provider.send(intent, approved_payload)
except DefinitelyNotSent:
    record_failed_before_dispatch(intent)
except AmbiguousTransportFailure:
    record_unknown_if_possible(intent)
    # The earlier durable intent remains if this recording fails.
else:
    record_provider_receipt(intent, receipt)

A real adapter must define how it knows DefinitelyNotSent. Categorizing every connection error that way would erase the distinction. Error wrapping, SDK retries and proxies all belong in that review.

Preserve ambiguity when final recording fails

In the jobs worker, only explicit Success and FailedBeforeDispatch remain those outcomes after Execute. Other values become Unknown. A canceled work context also forces a conservative unknown result.

There is an important limit: FinishEffect uses that work context. If cancellation prevents its database transaction, the intended final unknown update might not commit. The previously persisted pending effect still holds the operation out of ordinary retry.

This is why a durable intent and a final outcome are different records in the reasoning. The final record can be absent even when the effect happened. A system that requires a success row to recognize possible acceptance will resend work in precisely the crash window it needs to handle.

Expired claims with pending or unknown effects become visible as unknown in the jobs claim path. The state remains an operational responsibility, not a completed recovery merely because the type exists.

A bounded finalizer is different from background execution

Our inference gateway makes a different recording choice. When streaming ends or fails, it cancels the upstream context and creates a bounded detached context to record the durable outcome.

That context is for finalization only. The comment explicitly says it cannot redispatch or release a potentially spent hold. This distinction matters: detaching cancellation to finish accounting is useful; detaching arbitrary work can defeat the user's attempt to stop execution.

The gateway maps a stream error to outcome_unknown and unknown usage. It holds the terminal completion event until durable finalization succeeds, then forwards that terminal event. This is a completion-order boundary, not a claim that all earlier stream bytes were buffered or protected by a whole-response security gate.

If finalization fails, the gateway reports a fixed unknown-outcome error where it can. It does not echo arbitrary upstream error text into the response. Recording can still fail after the client disconnects; the design needs an operational way to discover and reconcile the durable attempt.

This gives a useful API principle: completion signals should correspond to the durable state the product claims, while partial response delivery remains a separate observation.

Keep the budget hold tied to evidence

A timeout can leave supplier liability unresolved. Releasing a spend hold simply because the caller stopped waiting would make that capacity available for another operation while the first may still incur cost.

Our ledger ResolveTx method permits evidence-free release only while the hold is reserved. Once dispatching or outcome_unknown, resolution needs authoritative final usage or explicit no-cost proof from the configured decision interface.

It validates the returned usage against the quote and route-related decision metadata. Repeated terminal resolution checks a digest and rejects conflicting evidence. These are implementation facts, not a financial audit or proof that every provider supplies complete evidence.

The gateway's cleanup path delegates ReleaseBeforeDispatch. Its safety relies on the backend and ledger state guard: an ambiguous dispatch commit that already advanced the durable state must not be undone by generic cleanup.

A simplified SQL-shaped rule illustrates the key condition:

UPDATE holds
SET state = 'released'
WHERE operation_id = :operation
  AND state = 'reserved';
-- Require the expected row count and commit the accounting changes together.

This sketch omits the real ledger postings, authority, scope, quote and evidence checks. The mechanism is the state condition, not a recommendation to mutate a hold without accounting.

Cancellation removes future permission

Cancellation can stop work that has not crossed the external boundary and can revoke a worker's publication authority. It does not erase accepted provider work.

Our jobs Cancel path advances generation and clears the lease for queued or running jobs while preserving pending and unknown effect records. The gateway requests upstream cancellation, but the runtime design does not assume the supplier necessarily stops or reports final usage immediately.

A user-facing status may therefore need two dimensions: execution canceled and external outcome unresolved. That is more honest than “nothing happened” and more actionable than an unexplained generic failure.

The same separation applies to approval revocation. It can prevent future dispatch if the adapter observes it in time. It does not retrospectively unsend an action a provider already accepted. Recovery needs to state what was prevented, what was sent and what remains unknown.

Reconciliation needs an evidence contract

The strongest recovery path depends on what the provider exposes. A durable receipt lookup can establish acceptance and terminal outcome. An authoritative object readback can identify the created effect. A usage feed can establish cost while leaving application success uncertain.

Each source has limits. A eventually consistent list returning no object is not necessarily proof of absence. A missing usage row may mean reporting delay. A matching message body may be insufficient to identify one operation among repeated legitimate messages.

Prefer a provider operation or receipt identifier with a defined status contract. Bind it to the local operation, tenant and adapter revision. Authenticate callbacks, deduplicate them and account for events that arrive before local mapping commits.

The broader integration request calls for verified webhook events to be persisted before acknowledgement and for early events to be replayed after acceptance mapping. That is a requested integration contract. This article does not claim a complete provider reconciliation service already satisfies it.

When a provider offers no suitable reconciliation or idempotency mechanism, the exposure policy must be tighter. The product can limit consequential operations, require review or choose a different integration. It cannot repair a missing external guarantee by renaming retries.

Bound operational limbo without inventing an outcome

Unknown cannot be a hidden permanent state. Give unresolved work an owner, age, evidence references and escalation policy. Monitor the number and age of unresolved operations separately from ordinary queued work.

A timer can expire a worker lease, schedule another status query or page an operator. It cannot convert ambiguous remote acceptance into definite nonacceptance. If the business chooses to absorb a loss or release customer-facing credit after a deadline, record that as a risk decision with accounting treatment, not a historical claim that the provider did nothing.

Bound the reconciliation workload as well. Unrestricted polling can overload a struggling supplier and fill logs with sensitive data. Use explicit retry budgets for read-only status checks, backoff, rate limits and an escalation owner. A status query that triggers work is not a harmless reconciliation API and needs a different review.

The objective is finite operational handling with honest knowledge. A resolved financial obligation and a known execution outcome can still be different facts.

Inventory every place that can retry

A workflow engine can recover unfinished steps. An SDK may retry transport errors. A gateway may fail over. A user can click again. An operator can replay a job. Each layer can turn an unresolved operation into another remote call unless the durable state is consulted at the effect boundary.

The runtime design explicitly disallows automatic inference resubmission after dispatch in its initial contract. A later explicit user request is a new operation with its own exposure and authorization, rather than an invisible exception to the first operation's ambiguity.

This does not say all APIs everywhere must forbid retries. A verified provider-native idempotency contract may support safe repetition. The important requirement is to prove the contract for the actual route and avoid letting a generic retry policy override it.

A trace should record why a new call was permitted. Was the old operation definitely never dispatched? Was the provider key still valid and bound to the same effect? Was this an explicitly new request? That evidence is much more useful than a retry counter alone.

Keep response, usage and effect evidence separate

A useful operation record can preserve three observations without forcing them to agree prematurely. Response delivery describes what the client received. Execution outcome describes what the provider established about its work. Settlement describes whether the liability has been resolved under the accounting contract.

Consider an illustrative response that completes before its usage observation arrives. The client can have a complete answer while settlement remains pending. Now consider a disconnected client followed by a valid usage observation. The system may know there was cost without knowing whether the user received useful content. Neither case fits a single success flag well.

Record observations as evidence with provenance rather than repeatedly overwriting the operation's history. If later authoritative evidence corrects an earlier estimate, preserve the revision relationship. The ledger's terminal digest conflict is a reminder that a correction needs an explicit contract; arbitrary late data should not silently mutate a settled result.

Privacy affects recovery too. A status interface may expose operation identifiers and typed outcomes without replaying the original private response. If response bodies are not retained, say so and give the caller a state they can act on. Do not regenerate a missing response by quietly invoking the provider again under the same unresolved operation.

This is proposed API structure around state mechanisms. It requires its own integration and user-flow tests. The purpose is to prevent the recovery interface from creating another effect while trying to explain the first one.

The fake-provider exercise

The package includes a local teaching model in which a fake provider appends an acceptance record and then raises a timeout. In the guarded sequence, the caller records uncertainty, does not resend and resolves using provider-side readback. One acceptance remains.

The fault variant blindly sends again after the same timeout. The fake provider has no native idempotency and records two acceptances. The local state remains unresolved rather than arbitrarily selecting one receipt as the correct operation.

Those are observations from a tiny synthetic exercise, not a measured duplicate rate or an end-to-end product result. Its value is that readers can run it, change the rule and see which oracle detects the error.

The real integration experiment should place failures at actual boundaries: before dispatch, after durable dispatch, after provider acceptance, after receipt arrival and during finalization. Include a definite pre-dispatch failure control that does allow the intended recovery, so the system cannot pass by holding everything forever.

The decision I would put in the API contract

Every consequential operation should have a durable identity, an explicit dispatch boundary, a place to retain uncertainty and a documented evidence path for resolution. Its budget state should follow that evidence. Its cancellation response should say what stopped and what remains unsettled.

Our code demonstrates useful parts of this contract and exposes the integration work still required. That is a more valuable result than calling every interrupted call failed or promising exactly-once effects across unrelated systems.

Walk one operation through a provider acceptance whose reply is lost. Ask what each retry layer sees, what the user sees, whether the hold survives and who resolves it. The full answer is a recovery architecture your agent can operate inside without guessing what happened.

Keep that contract visible wherever an agent can dispatch consequential work.

How this was verified

This article follows our gateway, jobs, worker, finance-adapter and ledger code, with the broader runtime architecture as design context. It does not claim every reconciliation connector is integrated or that a real provider has passed an end-to-end recovery test. The accompanying exercise uses a fake provider to make the distinction reproducible without creating real effects.