Trace agent decisions without turning telemetry into a private-content store

Our job-event digest binds five fields and deliberately excludes two others: delivery identity and collection time.

That distinction is small enough to fit in one function, but it determines whether a replayed delivery is recognized as the same semantic work. It also gives me a useful starting point for agent observability: retain the links that explain a decision, without treating every prompt, response or provider error as necessary diagnostic data.

The retained Event declaration has seven fields. QueueView exposes eight aggregate fields. The effect vocabulary names three outcomes. Those contracts are concrete. The broader enterprise tracing design adds policy, origin and transformation evidence around them, with content capture treated as a separate permission.

I want an operator to reconstruct what happened, identify what remains unknown and find the owner of the next decision. A transcript dump is often a poor substitute for that architecture. It can contain much more private material while still omitting the identity that would let us reconcile the actual effect.

Start with the question the operator must answer

An agent action can cross a request, a queued job, a worker attempt, a provider call and a final publication. When the user says the result is wrong, the operator needs to connect those stages without guessing which of several similar runs produced it.

I start with concrete questions. Which operation did the caller request? Which attempt owned execution? Which authority and policy revision applied? Did dispatch happen? Was there a definite result or an unknown outcome? Which observation established the final state?

Those questions select the necessary links. They do not automatically require the full content that flowed through the operation. A provider response body can be useful in a separately authorized diagnostic capture, but it should not become the default representation of every failure.

This is also an ownership decision. The team maintaining telemetry needs a contract for which questions it must answer, who may ask them and which evidence is available. Otherwise the easiest implementation is to keep everything and hope a future investigator can interpret it.

Separate operation, attempt and delivery identity

An operation identifies the intended business work. An attempt identifies a particular execution of that work. A delivery identifies a transport event that may be repeated or replaced.

The jobs code keeps those concepts separate. Event carries OperationID and ID. A claimed Lease adds AttemptID and fencing information. That gives retry and publication logic identities appropriate to their boundaries rather than one generic request string stretched across every purpose.

For tracing, I want those relationships retained. If a reply disappears, the operator should follow the same operation to its attempt and observed effect. Creating a new operation merely because the delivery was uncertain would hide the very connection needed for reconciliation.

A model session or conversation identifier answers another question: where the agent worked. It should be linked where useful, but it does not replace the business operation or grant permission to inspect all related conversations. Correlation is a diagnostic relation, not a disclosure rule.

Bind the semantic metadata deliberately

The actual digest function is short:

func Digest(e ports.Event) string {
	b, _ := json.Marshal([]any{e.Version, e.Type, e.Tenant, e.OperationID, e.Epoch})
	h := sha256.Sum256(b)
	return hex.EncodeToString(h[:])
}

Version, event type, tenant, operation and authority epoch contribute to the digest. Changing any one of those changes the expected identity. The ordering is part of the retained encoding contract; a new encoding cannot silently inherit the old meaning merely because it contains similar fields.

Delivery ID and Recorded are excluded. A replay arriving with another delivery identifier or a later collection timestamp should not become new semantic work on that basis alone. The exercise checks both exclusions independently and checks sensitivity to all five included fields.

The function itself does not validate the Event. Validation is a separate function. Nor does the digest bind an arbitrary domain payload that Event never contains. Those separations matter when another component tries to use a metadata match as evidence for a stronger conclusion.

A metadata digest cannot authorize a different payload

The source comment makes the boundary explicit: domain-specific intents must bind their own digest.

Two requests could share tenant, operation, type, version and epoch while differing in a recipient, attachment or amount outside this Event declaration. Their metadata digest would remain the same. That is not a cryptographic collision. It is the expected result of hashing the same selected metadata.

The domain owner must therefore define the canonical intent and its authorization binding. A notification intent needs the approved destination and content relationship. A financial intent needs its own money and funding contract. The queue can carry and reconcile identified work without becoming the authority that decides every domain meaning.

For an observability system, I retain which binding was checked and the assurance of that observation. A digest match can establish consistency with an expected metadata identity. It cannot establish that a human approved the exact payload or that the final effect used current authority.

Metadata-only is a schema property, not anonymity

The Event type has no arbitrary prompt, response or provider-body field. That is a useful restriction on the queue interface. It narrows what a well-behaved producer can casually attach to a durable event.

The validator also bounds identifier syntax. Its token expression permits one to 120 characters from an explicit set. Event type has a separate versioned grammar and a 120-character length ceiling. Epoch must be positive, and the accepted Event version is one.

Those checks do not make identifier values anonymous. In the synthetic exercise, customer_alice is a valid tenant token. It satisfies the syntax while plainly carrying information a real system might consider identifying. An opaque identifier can also remain sensitive through linkage to another store.

I therefore treat schema minimization, identifier generation, access control and retention as separate responsibilities. A producer needs rules for what belongs in an identifier. A trace reader needs authorization. A retention job needs to cover metadata, not only bodies. The validator cannot supply those policies by accepting a string.

Keep high-cardinality identity out of metric labels

The queue view gives operators bounded aggregates: queued, running, poison, unknown, cancelled, epoch-held, oldest age and maximum attempts.

That is a different interface from the event trace. The view can reveal a growing unknown backlog or aging work without using operation and tenant identifiers as telemetry labels. The SQL is scoped to the caller's tenant, while the returned QueueView declaration contains aggregate fields rather than those identifiers.

I keep the detailed identity links in an appropriately authorized trace store. Metrics answer aggregate operating questions. Traces answer which operation or attempt produced a particular state. Combining the two by putting every operation ID into a metric label grows the series space with the work and makes access and retention harder to reason about.

The pure exercise checks the declared QueueView field contract and numeric representation. It does not execute the view SQL or prove tenant isolation in a database. The architectural rule remains useful: choose the smallest aggregate that answers the dashboard question, and make detailed correlation an intentional lookup.

Use typed outcomes instead of provider-body storage

The effect vocabulary names success, failed_before_dispatch and unknown. The persistence method checks accepted outcomes before updating effect and job state. The worker also normalizes unexpected handler outcomes to unknown, and cancellation after possible dispatch cannot manufacture a definite pre-dispatch failure.

That design keeps uncertainty operationally visible. A lost acknowledgment is not automatically a retryable failure. It also avoids making an arbitrary provider error body the durable state label that every queue viewer or metrics consumer inherits.

A typed category will not answer every diagnostic question. The system may need a bounded error classification, provider request reference or separately authorized evidence packet. The important choice is to define those additions as part of a schema with an owner, rather than allowing arbitrary response text to become the default trace payload.

The pure declaration exercise checks the three named constants. The original jobs unit suite supplies additional worker-boundary evidence, including lost receipt, ambiguous result and cancellation cases. I keep those scopes distinct from a full database or provider integration.

Store observation assurance with the claim

A trace event can contain a statement from the actor, a verified service response or an independent readback. Those are different assurance levels even when they describe the same operation.

In the enterprise design, I retain origin assertions and their assurance, policy and configuration revisions, transformation lineage and permitted digests. This gives a future reader a way to distinguish what a component claimed from what another boundary actually established.

A proposed observation envelope might carry the following metadata, with its own access and retention contract:

{
  "operation_id": "opaque-operation-id",
  "attempt_id": "opaque-attempt-id",
  "policy_revision": "policy-revision-id",
  "outcome": "unknown",
  "evidence_class": "provider_ack_missing",
  "assurance": "source_assertion"
}

That is a design example, not an additional deployed Event schema. The finite evidence class explains the uncertainty without copying the provider response. A later independent observation can be linked as another record rather than overwriting the original assertion with a stronger story.

Integrity links do not establish truth

An append-only sequence or signed envelope can help detect certain modifications and missing links. It does not prove that the original statement was factually correct, that a human authored it or that every relevant action passed through the recorder.

I want the report to retain that coverage boundary. An actor assertion signed by the collector is still an actor assertion with a protected record. It does not become independent acceptance evidence because the signature verifies.

This also affects how I investigate incidents. A gap in a chain is useful evidence about the record. A complete chain is useful evidence about the links it covers. Neither replaces the external observation needed to establish whether the payment occurred, the permission changed or the intended record persisted.

The tracing architecture should help the operator ask the next question. It should not upgrade the assurance of a claim simply because the storage format is cryptographically tidy.

Hashing a small secret does not make it private

A plain hash can reveal a low-entropy value to someone who can enumerate its likely inputs. The algorithm does not have to be broken.

I included a tiny synthetic demonstration using three plan labels: basic, plus and pro. Hashing those three candidates creates a dictionary that identifies all three corresponding plain SHA-256 values. The experiment concerns those explicit choices, not a real customer record or a measured production leak.

The enterprise design calls for keyed, domain-separated digests where sensitive content digests are permitted. That has its own obligations: key custody, authorized verification, domain choice, rotation and deletion behavior. A keyed value can still be linkable, and a consumer with the key can compute guesses. I do not call that anonymization either.

I also keep content-digest policy separate from operation identity. Rotating a diagnostic hashing key should not quietly create permission to repeat an unresolved business effect. The operation and its reconciliation contract need a stable identity appropriate to that job.

Preserve transformation lineage without preserving every body

Agent workflows increasingly enrich, redact, classify and transform information before the final response. If an operator only sees the final text, it can be difficult to determine which stage introduced a decision or removed a condition.

The enterprise design gives those stages their own action and attempt identities, parent links, policy revisions and transformation types. A permitted digest can relate inputs and outputs where the capture policy allows it. Read projections and destination restrictions remain separate controls on what each stage can actually receive.

That produces a useful debugging path without assuming every intermediate body must be stored. The investigator can see which transformation ran under which contract and locate the authorized evidence needed for a deeper examination.

The tradeoff is real. Content-free lineage cannot reconstruct content it never retained. If the product requires that capability for a particular incident class, the owner must define an explicit capture purpose, scope, access rule and lifetime before relying on it. A correlation ID cannot recreate omitted evidence later.

Give capture, access and retention different owners

Permission to process content does not automatically grant permission to keep it in a diagnostic store. Permission to keep a record does not automatically grant every operator permission to read it. A private enterprise flow does not become contribution or commercial data because it shares a correlation interface.

I keep those decisions explicit in the enterprise architecture. The request and action trace has an operational purpose. Optional history, diagnostic capture and other uses need their own permissions and storage behavior.

For an operations team, the practical artifact is a field inventory. For every field, record the diagnostic question it answers, its source, allowed reader, retention boundary and deletion owner. Include metadata identifiers and digests in that inventory. They are often the links that make the rest of the information identifiable.

That work is more useful than a blanket instruction to avoid sensitive logs. It gives implementation and review a set of concrete conditions to check when another tool, model or exporter is added.

Build the smallest trace that answers one incident

Choose one real operating question, such as why an operation remained unknown after a worker lost its reply.

Follow the operation to its delivery, ownership attempt, authority epoch and typed result. Identify which observation established each fact and which component can reconcile the remaining effect. Then ask whether every retained field was necessary for that investigation.

If the answer requires a body that was not captured, define that as a separate diagnostic capability rather than quietly broadening all telemetry. If the answer is already available from identities and typed evidence, keep the body out of the default path.

The leadership decision is about useful observability capacity. Operators need enough evidence to make the next correct decision, and clear authority to obtain more when necessary. The best trace is the one that answers the question with a defensible data contract.

How this was verified

Eight declarations are retained verbatim from OptinRouter commit 88eb34a7d635ce8076b8a06ca90d5e451ce4b78b, with Go-AST byte ranges and hashes. Package/import headers and the minimal ports namespace are fixture adapters. The pure Go 1.22 exercise runs 60 metadata, digest, syntax and field-shape cases. Three author mutants compile successfully and then panic on assertions; restoring the source exits zero. Independent review is pending.

The original jobs unit tests also pass: three top-level tests, including eight worker failure-boundary subcases. A syntax-valid missing-epoch digest mutation makes the original metadata test fail; restoration passes. This replay uses cached Go 1.26.9 with only the original module's 1.27.2 language directive adapted, preserving dependency versions and production/test bytes. PostgreSQL integration is excluded.

The three-choice hash demonstration uses synthetic labels and a known fixture key. It is not a production privacy incident or a cryptographic security proof. The packet does not establish a complete deployed trace exporter, anonymous identifiers, metadata retention/deletion enforcement, database authorization or a domain-payload approval binding. The broader enterprise tracing layer remains a design contract around the narrower implemented jobs model.

The independent set was pinned at content-video 313cecb before the reviewer read the exercise. The initial 60-case exercise killed 23 of 24 runtime mutations (receipt ec7fac3). Two additional API-shape changes were detected by compile-time consumer contracts and are reported separately, not counted as runtime passes. Adding the single-segment rejection and matching valid multi-segment case expanded the exercise to 62 cases. The same-set follow-up killed 24 of 24 runtime mutations, retained both compile-contract detections, and returned a clean control exit of zero (receipt df3787b). Source declaration hashes stayed identical; this is a follow-up, not a fresh independent score.