Fourteen cuts through a streaming security boundary

I split a 15-byte fake canary into two chunks at every internal boundary. The chunk-only checker released the complete marker in all 14 cases. A cumulative checker caught the completed marker, but a fragment had already escaped in every case.

Whole-unit buffering released none of it. A rolling window also released none under this one fixed-length pattern rule. All four implementations still passed the ordinary-output control.

This small exercise changed the way I frame streaming security. The question starts with the unit the system promises to protect and the moment that unit becomes observable. Detector quality matters after those semantics are clear.

I built our policy composer around one bounded buffered unit, and our enterprise design explicitly separates buffered and streaming delivery. Here is the reasoning I would carry into any assistant that inspects output, streams a response or assembles a tool call.

The experiment is about release, not model intelligence

The marker is DEMO_CANARY_7Q9. It is synthetic and has no credential value. There are 14 places to split its 15 bytes into two nonempty pieces.

The first policy checks each piece independently, then emits it if the complete marker is absent. Neither piece contains the full marker, so both pass. The receiver joins them and obtains the marker the policy was meant to withhold.

The second policy checks everything seen so far before emitting the newest chunk. It detects the complete marker when the second chunk arrives. It cannot retrieve the prefix it already emitted. In these cases it released a fragment rather than the whole marker. That is a distinct observation, and I do not label it a complete credential leak.

The third policy collects the whole unit and checks before releasing any of it. The fourth retains a lookbehind window of marker-length minus one. That works for this bounded substring rule because every incomplete prefix fits within the retained suffix. It says nothing general about semantic rules whose decisive context may occur much later.

The useful comparison is four release contracts, not four model scores. The accompanying Python program uses no provider, no customer content and no product runtime. Readers can run the mechanism and its fault control locally.

Name the first observer

Before choosing a stream design, I ask where output first becomes visible. A browser is one observer. A proxy, analytics hook, log sink, tool adapter or automatic resource loader may see it earlier.

If a required inspection runs after one of those boundaries, its prevention claim must stop before that observer. Masking text in the browser does not retract the raw text already sent over the network. Hiding a message after rendering does not retract an image request the renderer initiated.

The diagram therefore needs more than provider, classifier and user. It needs the buffers, callbacks and sinks between them. Mark every copy or forwarding point that can expose the protected bytes.

I treat those paths as product behavior to test. A server callback returning success is not a complete oracle for whether a browser, telemetry system or external destination received something. Use a receiver-side observation when the claim is about delivery.

A transport chunk is not a policy unit

Chunk boundaries are often arbitrary from the application's perspective. They can separate a word, an escape sequence, a structured field or a multibyte character. A network read boundary does not establish that the resulting piece has independent meaning.

The exercise includes both internal splits of a three-byte UTF-8 euro sign. Decoding each byte chunk independently fails in both cases. An incremental decoder reconstructs the character in both. Those are concrete parser observations, not a guarantee about every upstream protocol.

The same category appears above the byte layer. An incomplete JSON object is not yet a validated tool argument. A sentence boundary does not necessarily end a confidentiality dependency. Two individually ordinary statements can reveal something sensitive when combined.

Choose the unit based on the property the check can establish. A bounded token pattern may support a finite window. A whole-document schema needs the whole relevant structure. A semantic disclosure decision may require the question, source restrictions, intended audience and broader response.

Compare the three delivery choices honestly

Full buffering gives required checks the complete selected unit before any of it leaves. It adds waiting and memory pressure, but its temporal promise is clear. It does not make the detector infallible; it makes the detector's decision precede delivery of the unit it checked.

Unit buffering releases independently checked frames, sentences or records. It can improve responsiveness where the policy is truly local to each unit. The independence condition is the difficult part. An instruction or disclosure spanning units can defeat a rule that assumes each piece is self-contained.

Unrestricted streaming optimizes immediate delivery while limiting what a later whole-response check can prevent. It can still have local format rules, scoped tool permissions and monitoring. The product should describe those protections accurately rather than implying that a final moderation decision protected earlier bytes.

I do not treat these as a universal ranking. A public answer selected from approved paragraphs has a different risk surface from a private diagnostic summary or a consequential tool call. The delivery contract should follow the workflow and its restrictions.

Make final-byte validation part of composition

Our Go composer accepts a bounded buffered unit. Its implementation rejects units above 1,048,576 bytes and accepts deadline values from 1 through 120,000 milliseconds. These are code limits, not measured capacity or a promise that a user should wait two minutes.

It checks current authority, runs ordered controls, validates configured grammars again after transformations and evaluates current authority again before returning protected bytes.

That order matters because checks compose over changing objects. A redactor can change structured output after an earlier grammar accepted it. A repair model can produce a different artifact. A lease can expire while inspection runs.

The protected decision should describe the actual final artifact at its release boundary. Earlier success is not automatically transferable to later bytes.

Illustrative structure, not a drop-in API:

def protect_unit(raw, policy, authority):
    require_current(authority)
    data = bounded_copy(raw)
    data = run_required_controls(data, policy)
    validate_final_grammars(data, policy)
    require_current(authority)
    return ProtectedUnit(data, policy.revision)

The effect adapter still owns its own authority boundary. Returning bytes from a composer does not make a later external write atomic with that function's check.

An async check needs an honest job description

Some inspections can run after delivery because their purpose is analytics, sampling or later investigation. Required prevention checks cannot claim that same placement.

Our enterprise action design assigns required or optional status to branches and requires the mandatory branches to join before their protected release boundary. A slow mandatory branch does not become optional because another branch finished first.

The merge rule also needs to be deterministic. Arrival order is a poor substitute for policy. A timeout should produce the configured block, hold or error state with an owner, rather than silently removing the missing result from the join.

This is where latency optimization belongs: execute independent checks concurrently, reduce unnecessary input projection and avoid duplicate work while preserving the required join. Moving a mandatory check after delivery changes the protection contract. It is not merely a faster implementation of the same guarantee.

Tool calls need their own release boundary

Streaming text and streaming an action proposal are different operations. A partial tool argument should not become a partially authorized effect.

An adapter can collect a complete typed call, validate its schema, resolve current authority and bind any required approval before execution. The user may see progress while the actual side effect remains gated. Progress text should not imply that the effect has already committed.

Model-generated destinations also need separate policy. A harmless-looking partial URL may become a different destination when later bytes arrive. A renderer or speculative fetch should not act on incomplete model text before the destination policy sees the final resource request.

The same principle applies to structured fields that arrive in separate events. Validate the complete protected structure and its semantic effect facts. A syntactically valid field fragment does not establish authorization for the assembled action.

This design separates useful responsiveness from premature execution. The assistant can explain progress without the broker accepting an incomplete operation.

Budget for the gate, not just the answer

Inspection and repair consume resources. A product that spends its entire deadline and inference budget producing the draft can leave no room for required output validation.

Reserve the relevant classifier and repair work before dispatch. Bound retries, repair attempts and graph fanout. If a mandatory stage cannot run inside the remaining budget, enter the intended typed state rather than quietly disabling protection.

Repair should not become an unbounded loop in which the model repeatedly rewrites a rejected answer. Every new artifact needs the checks appropriate to its final bytes. Reusing the original approval or grammar result after a material rewrite would sever the evidence relationship.

We had a concrete warning about inspection latency in the local safety-model work. In the11 October CPU experiment, Qwen3Guard Gen 0.6B took about 4.5 seconds at the median on 120 synthetic safety rows, with four threads per worker on a shared host. It flagged 60 of 60 unsafe rows and one of 60 allowed rows under that experiment's mapping. The run did not include an integrated streaming gate.

That result gives me a real stage to budget and measure. It does not establish that buffering adds exactly 4.5 seconds to a product request: generation can overlap some work, queues can add delay, and a final whole-unit decision may still wait for completion. I want the actual critical path measured rather than summing unrelated medians.

Do not report a numerical latency tax without measuring the actual route. Generation and inspection can overlap in some designs, while a whole-document rule may require serial work. Queueing and model placement can dominate the observed delay. The correct comparison is the workflow distribution under the chosen release contract.

Cancellation does not recall released data

When the user cancels, stop descendants and unused work where the runtime can enforce it. Record what was already released or committed. A cancellation acknowledgment should not imply that earlier disclosure disappeared.

For an upstream provider call, stopping local delivery does not prove the supplier stopped processing. Outcome and budget reconciliation remain a separate responsibility, as in any external effect whose reply can be lost.

For a buffered unit that has not crossed the release boundary, cancellation can prevent delivery. For a partially released stream, the system can prevent additional bytes while retaining an honest record of the prefix that was observable.

Those are different outcomes worth representing in the API. A status can distinguish withheld, partially delivered, completed and unresolved. A single generic error often obscures the fact the user or another sink already received material.

Backpressure is a security and availability decision

A slow inspection stage can make the buffer grow. Without bounds, a policy gate can become a memory-exhaustion path or an indefinite wait.

Set byte, nesting, token and time limits for the protected unit. Decide how upstream production responds when the buffer is full: pause where supported, terminate the attempt or hold under a bounded policy. Do not spill private content into an unreviewed temporary store merely to preserve the appearance of streaming.

If a hold persists for review, give it an owner, expiry for operational handling and a recovery path. That timer can trigger a refusal or escalation. It does not authorize releasing uninspected data because the queue became inconvenient.

The original producer, inspection worker and downstream connection also have different failure clocks. Track them separately so a stalled client does not hold an inference process, memory allocation or database transaction indefinitely.

Measure the timeline you are promising

I would instrument generation start, first provider byte, protected-unit completion, required-check completion, release start, first receiver observation and terminal status. Keep timestamps attributable to one attempt and policy revision.

For each mode, report time to first permitted output and time to complete permitted output, plus buffering volume, refusal rate, cancellation behavior and legitimate task success. A faster first byte can be a worse result if it is outside the claimed protection contract.

If clocks cross machines, record the uncertainty or use local intervals and causal ordering. A neat millisecond diagram should not imply synchronization the system does not have.

The comparison should include failures: oversize units, unavailable classifiers, malformed encodings, incomplete structures, client disconnects and an authority change during inspection. Successful-only latency can hide the cases that consume the most operator time.

Put the observer in the test harness

The small exercise establishes an ordinary-output control,14 marker splits, a disabled-check fault and two UTF-8 split cases. That makes the release rule independently reproducible at the toy-policy level.

For a real route, put the observation after the actual transport and renderer. Verify whether data reached the client or triggered a resource request. Run the same adversarial unit through alternate output paths, including debug streams, tool previews and error handling.

Vary the transform order. Start with a valid structure, alter it after its first validation and ensure the final release check detects the new invalid artifact. Remove a required branch and verify the configured refusal. Expire authority while a slow stage runs and check the next committing boundary.

These tests need allowed controls. A route that withholds every ordinary answer can look good on a leak-only metric while failing the product. Keep useful completion and false interruption in the same review.

Preserve the sequence when policy changes

A new policy revision can arrive after a unit was generated but before it is released. Pin the generation context for explanation, then define which current restrictions apply at release. A permissive result under an obsolete revision should not automatically survive a critical revocation.

The release owner needs to know which bytes were checked, which revisions produced the result and whether another transform or retry changed the artifact. Cached inspection results require the same identity discipline. A result for one tenant, source version or release mode should not silently authorize another.

This adds metadata work, but it makes the eventual question answerable: why did this unit leave under these conditions? The corresponding acceptance experiment delays a required stage, changes authority and then lets the old result return. Observe the actual sink rather than assuming the original result still applies.

What I would put in the product contract

Define the protected unit, its maximum size, the observers before and after the gate, the required checks and what happens when they fail. State whether streaming is full, unit-buffered or withheld until completion. Make partial delivery visible when it occurs.

Then measure that contract on the actual deployment profile. Local inspection, hosted inspection and a browser renderer have different placement and privacy implications. A certified buffered path does not automatically certify a streaming adapter.

My default for a new sensitive workflow is to start with one bounded buffered path whose behavior I can explain. I would relax it into smaller release units only when the check's scope and the measurements justify that tradeoff.

The exercise is deliberately small:15 bytes,14 splits and four policies. Its useful lesson is durable. The system should know which bytes it has permission to release before it asks how quickly it can show them.

How this was verified

The recorded exercise uses a fixed fake marker and standard-library Python; it measures delivery behavior, not semantic detection quality. The code facts come from platform/policy/compose.go. The enterprise action and streaming rules come from the 18 October architecture chapter and remain design requirements where not integrated. No new product-runtime, provider or throughput test was run. The protected-unit size and deadline are implementation limits, not measured latency or production certification.