Durable agents need agreement between delivery, ownership and effects

I want a worker that resumes after losing its lease to finish useful computation without regaining the right to publish. That is why I separate recovery from publication authority.

That problem becomes more visible as agent systems grow beyond a single interactive session. A task may wait on a model, a browser, a human approval or a remote service. Processes restart. Requests are redelivered. A cancellation arrives while an external action is in flight. The queue still has work, but the system needs to know which work remains legitimate.

Our October development work put several related mechanisms beside each other: transactional enqueueing, consumer inboxes, finite leases, publication fences, explicit effect records and an append-only runner journal. The important insight was how much each mechanism leaves for the next boundary to enforce.

The jobs module is implemented but remains unregistered at its integration checkpoint, with external effects disabled. I treat composition and real PostgreSQL evidence as separate work.

Begin with the domain transition

Imagine a fictional report-generation workflow. A user submits an authorized request. The application records that submission and schedules analysis. If it commits the submission and crashes before scheduling, the report can remain forgotten. If it schedules first and the submission rolls back, a worker may analyze work the domain never accepted.

The outbox pattern puts the accepted domain change and its delivery intent in one transaction. A relay later delivers committed intents. Duplicate delivery remains possible, particularly if delivery succeeds and the relay crashes before recording the acknowledgement.

Our Enqueue method accepts the caller's transaction and does not commit it. That API shape is consequential: the domain owner can keep the domain change and enqueue in the same commit boundary.

Here is an illustrative SQL transaction, not our production schema or a migration:

BEGIN;
UPDATE submissions
SET state = 'accepted'
WHERE tenant_id = :tenant AND id = :submission;
INSERT INTO outbox(tenant_id, event_id, operation_id, event_type)
VALUES(:tenant, :event, :operation, 'report.accepted.v1');
COMMIT;

Real code must also verify affected rows, authority, input binding and uniqueness. The snippet shows the atomic boundary rather than claiming those omitted requirements are optional.

Two components using PostgreSQL do not automatically share a transaction. If a workflow engine cannot accept the caller's transaction under its pinned API, retain the outbox handoff and test it. AWS's primary guidance describes the transactional-outbox pattern and the need to handle duplicate delivery: https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html.

Give the operation a stable identity

Delivery identity and business identity serve different purposes. A relay can produce multiple deliveries of one event. A worker can make multiple attempts at one operation. A new attempt should not become a new business request merely because a process restarted.

Our jobs event carries a tenant, operation ID, event type, version and authority epoch. Its digest binds that semantic metadata. The code deliberately excludes delivery IDs, timestamps and private text. Domain-specific intents must bind their own material content or immutable reference; the generic event digest does not magically cover an arbitrary payload.

That last detail is easy to miss. “We hash the job” sounds complete until you inspect what the hash includes. If an operation refers to an outbound message, its content and recipients need a domain-level binding. A job's stable identity prevents certain duplicates; it does not establish what every downstream effect means.

The identity review should distinguish operation, event, delivery, attempt and external receipt. Record which scope owns each uniqueness constraint. A tenant-scoped identifier should not accidentally become a cross-tenant cache key or deduplication boundary.

Commit the inbox with the consumer effect

An inbox is useful when its marker and the consumer's database transition commit together. Marking a message processed before applying its effect creates a lost-work window. Applying the effect and recording the marker later creates a duplicate-effect window.

Our Inbox function receives the caller's transaction. It reserves a semantic operation for a consumer, checks its digest and separately records delivery identity. A repeated delivery with a changed operation or digest becomes a conflict. The returned boolean indicates whether the semantic operation was newly reserved, allowing the caller to skip an already applied operation within the transaction's contract.

This provides two valuable protections: transport redelivery can resolve to existing work, and semantic reuse with different meaning can be rejected. The caller still needs to commit the inbox and its own database effect together. An API that accepts a transaction gives the caller that capability; correct integration must actually use it.

Retention is part of this design. If the application deletes uniqueness records before its supported replay window ends, an old event can become apparently new. Financial or irreversible business identity may need a lifetime independent of broker retention.

Choose one owner for the queue lifecycle

The architecture considers a workflow adapter behind a port. The SQL jobs module is a related candidate implementation. They are not a reason to run two schedulers over the same logical work.

Our integration contract explicitly requires exactly one queue claimant and keeps the foundation probe from consuming it. “One owner” does not mean one worker process. Several executors can participate in one implementation. The restriction is on competing authorities for claim, retry scheduling, lease recovery and completion.

This matters during migrations as well as initial deployment. An outbox relay owns delivery acknowledgement; it need not become a second workflow scheduler. Moving between adapters requires draining or explicitly transferring work and its identities before activating the new owner.

The practical review is to list every loop that can make a job runnable. Include startup recovery, reapers, broker retries and administrative replay. If two mechanisms can independently reset the same operation, the lifecycle has competing authors even if the normal worker path looks clean.

A lease solves recovery, not stale execution

Our Claim method uses a short transaction with FOR UPDATE SKIP LOCKED. It selects eligible work, changes ownership, generates an attempt ID, advances the generation and sets a finite expiration.

SKIP LOCKED is appropriate for queue-like access where workers should avoid waiting on each other's claims. PostgreSQL documents that it yields an inconsistent view and is unsuitable for general-purpose consistency checks. It does not provide universal ordering or fairness. See the primary SELECT documentation: https://www.postgresql.org/docs/current/sql-select.html.

A lease allows a later worker to recover abandoned work. It cannot physically stop the old process. The original worker may be paused by scheduling, network loss or a long computation. After expiry it can resume with stale assumptions and a perfectly plausible result.

That is why the generation matters. Each new claim advances a value that the committing sink can compare. A stale worker's successful computation does not preserve its authority to publish.

Renewal also belongs inside the authority model. It should require the current owner and attempt under the current epoch. Otherwise a stale worker might extend authority that has already moved elsewhere.

Put the fence where authority becomes state

Our Store.locked check verifies tenant association, current authority epoch, job generation, owner, attempt ID, operation metadata, running state and lease expiry. Publish invokes that check, runs a database-only callback and checks again within the transaction boundary.

The point is to serialize publication with authority rotation and competing claims. A separate preflight check followed by an unrelated commit would leave a gap in which ownership could change.

Illustrative shape, not a complete copy of the implementation:

return authority.Within(ctx, principal, func(tx Tx) error {
    if err := lockCurrentFence(tx, lease); err != nil { return err }
    if err := publishDomainState(tx); err != nil { return err }
    return lockCurrentFence(tx, lease)
})

The callback must not perform external I/O. Holding locks across model inference or a provider call would turn a short authority transaction into a long resource dependency. More importantly, a database lock does not make a third-party service participate in the transaction.

For object storage, one useful proposed pattern is immutable result objects plus a fenced database manifest. A stale worker may leave an unreferenced object, but it cannot make that object the authoritative result if the manifest publication refuses its fence. Orphan cleanup is then a separate retention task.

Record intent before the external effect

An external message or billable inference call creates a different problem. The provider may accept it even if the worker never receives a response.

The jobs module records an effect as pending before execution. It only permits another beginning after a definite failed-before-dispatch state. Claim excludes work with pending, successful or unknown effects. Expired work with unresolved effects becomes visibly unknown rather than silently reentering the ordinary retry path.

The worker checks current authority around beginning the effect. It bounds execution by the configured timeout and actual lease. After Execute, only explicit Success and FailedBeforeDispatch remain those outcomes; other values become Unknown. If the work context is canceled, the outcome is conservatively unknown.

Persistence can itself fail. Because final recording uses the work context, cancellation can prevent FinishEffect from committing a new state. The earlier pending intent remains important evidence and continues to block blind redispatch. Calling the intended final state “unknown” does not mean a final unknown record always committed.

A provider-native idempotency contract or receipt lookup can improve recovery, but the exact scope, retention and payload binding must be verified. A SQL preflight cannot fence a remote provider that does not understand the fence.

Cancellation removes authority, not history

Our Cancel path advances generation and clears the lease on queued or running work. It preserves pending and unknown effect records.

This is a useful separation between a user's desire to stop future work and the unresolved consequences of work already sent. Canceling a job can invalidate a worker's publication authority. It cannot reverse a remote acceptance or prove that no liability remains.

The product needs to express both facts. “Canceled” may accurately describe future execution while the effect outcome still needs reconciliation. A single terminal status often hides this distinction and encourages an unsafe retry or refund policy.

Give unresolved effects an owner and an operational view. A hold without a resolution process can become an outage. A timeout on that process should escalate or invoke an explicit risk policy; it should not fabricate proof that the original request never happened.

Durability still depends on the storage boundary

The runner has a separate append-only WAL implementation with length framing, a CRC, file locking and fsync. Readers recognize a torn or corrupt tail, and repair quarantines the tail before truncating to the verified prefix. Appending after an unrepaired torn tail would otherwise make later records unreachable to normal replay.

That is a concrete local durability mechanism, not a replacement for the jobs database or a guarantee about external effects. A CRC detects accidental corruption; it does not authenticate a malicious writer. A local fsync does not create an off-host disaster-recovery copy.

The relationship worth preserving is conceptual: record meaningful intent and state before relying on it after a crash. Each storage system still needs its own recovery procedure, authority checks and failure assumptions.

Do not merge these implementations into a fictional unified runtime in the article or the product. Their boundaries and integration contracts are part of the result.

Test the transition that would make the old worker dangerous

A useful fence experiment first establishes that the current worker can publish. Then advance ownership and show the old worker cannot publish while the new one can. Without the allowed control, a broken fixture that rejects everything could look secure.

Other crash-boundary experiments roll back domain acceptance and enqueue together, redeliver a committed event, change a digest under the same operation, cancel after effect intent and lose the provider response after acceptance.

The test oracle should be authoritative domain state and provider-side evidence, not only a worker log. For PostgreSQL behavior, use the actual pooled database, concurrent transactions and deployed schema. A local state-machine demonstration can teach the mechanism but cannot establish PostgreSQL isolation or production registration.

For this editorial package, the executable exercise is explicitly a small local model. It makes the stale-generation and ambiguous-acceptance examples reproducible with fault variants. It is supplementary explanation, not a new certification of our module.

Recovery authority should survive a restore

A restored database can bring back a previously valid generation or permission. The broader runtime design therefore treats recovery authority as more than replaying job rows. Restriction evidence and authority epochs need a source that is not silently rolled back with the same stale snapshot.

This is a proposed recovery contract, not a property established by the jobs file alone. It matters because a correct live fence can become meaningless if disaster recovery reintroduces an old authority history and immediately enables writers. Keep effects frozen while reconstructing restrictions and validating the new authority epoch. Record what can safely resume and which unresolved external operations still require evidence.

Schema and event versions need similar discipline during a rolling release. A stable operation ID with a new interpretation can be as dangerous as a duplicate ID. Prefer additive changes, explicit versions and a compatibility window that includes recovery readers. A projection replay should have credentials that cannot issue external effects merely because it encounters historical action events.

The acceptance test should restore an older fixture containing previously valid work, then establish that newer restrictions still prevent it from publishing. This is a distinct experiment from pausing a worker. Both challenge current authority, but one changes a process and the other changes the recovered history.

Measure how long work remains unresolved

Queue depth alone cannot distinguish healthy retries from dangerous ambiguity. Useful operational categories include queued, running, poison, unknown, canceled and held under an old epoch. Age matters as much as count.

Track attempts and oldest unresolved age by tenant and workflow class. Distinguish lack of worker capacity from missing authority, invalid configuration and a provider outcome requiring reconciliation. Avoid logging raw prompts or provider bodies just to explain those states; identifiers, revisions and typed reasons often provide a safer starting point.

Before enabling effects, decide who reviews poison work, who can reconcile unknown work and what evidence is required to release a hold. Make authority to repair narrower than authority to execute arbitrary jobs. Replaying an event to rebuild a projection should not also send an irreversible action.

The implementation choice I would carry forward

Durable agent execution is an agreement between several boundaries. Domain acceptance and delivery intent share a commit. Consumer deduplication and database effects share a commit. Claims have finite ownership. Publication checks the current fence. External effects carry durable identity and retain ambiguity until evidence resolves it.

These are ordinary distributed-systems mechanisms, made especially consequential by long-running, probabilistic and tool-using work. The agent's intelligence can improve the proposed result. It does not settle who may publish or whether a timed-out request already happened.

Review one workflow from acceptance through a paused worker and a lost provider reply. Name the owner of delivery, scheduling, publication and reconciliation. That walkthrough will tell you more about durability than a queue's feature list.

How this was verified

This article follows our jobs, worker and runner code alongside the runtime design and integration requests. The jobs module is implemented, but its integration checkpoint says it remains unregistered pending queue-owner selection, schema allocation and real PostgreSQL evidence. External effects are disabled there. Those facts limit the claim: we can explain concrete implementation choices without pretending the entire deployment has been validated.