One per-session line broke the shared prompt cache

Two identical headless starts differed by one unique line inside a tool description. That line sat before the rest of the prompt, so every token after it stopped being reusable across sessions.

I was investigating why fresh Claude Code sessions appeared to rewrite most of their startup context. My first suspect was the per-session scratch directory: every run has a different path, and that path appears near the start of the first message. It was plausible, and the diff confirmed it changed. But it was not the only change.

I captured two request bodies from otherwise identical claude -p boots and compared them. The tool list came first in the prompt prefix. One tool description had an appended native transport binding, unique to the session. A later per-session path in the first user message also differed. Both introduced entropy; the tool-description marker occurred earlier, so it invalidated every following tool definition, system instruction and first-message token.

Why one byte can affect the whole tail

Prompt caching is prefix-based. The provider can reuse a cached prefix only while the incoming sequence matches the prior sequence. Once a difference appears, the suffix after that point is no longer the same prefix. If a changing value is near the beginning, the cache loses a large amount of otherwise stable content.

In this boot, tool definitions made up about 65K tokens, with the MCP tool descriptions accounting for much of that. The session-specific marker was embedded in one tool's description before the remainder of the tool list and system prompt. That meant the cache boundary moved forward for every session even though almost all the instructions and tools were identical.

This is a different kind of prompt cost than “the prompt is too long.” The problem was not an overly verbose static instruction. It was a small variable placed inside the stable prefix. Trimming the system prompt would have targeted the wrong layer and could have discarded useful operating context without restoring cache reuse.

A controlled comparison beats a plausible guess

The useful experiment was not “remove the scratch path and see whether usage improves.” That would have changed one suspected variable while leaving the earlier marker intact. Instead I inspected the two serialized requests and asked which bytes differed, where they appeared, and which client actually consumed each value.

That separated two mechanisms:

• the native binding belonged only to the Codex bridge, which reads it back through its own client path;
• Claude Code did not consume it, so including the value in Claude's tool definition only made the prompt unique;
• the SessionStart scratch path was a separate difference in the first user message and remained a separate cacheability issue.

I changed the server to append that binding only when the requesting client was the Codex app-server. The relevant code is a client capability check around one tool-description transformation:

const needsBinding = tool.name === "agent_channel_reply"
  && channelRuntimeHandle
  && codexChannelRequested;

return needsBinding
  ? { ...tool, description: tool.description + bindingForThisClient }
  : tool;

The binding itself stays private. The point is that a value should be serialized only for the recipient that needs it. A test checks that non-Codex clients receive the plain description while the Codex path still receives its binding; a control that restores the old condition fails the test.

Keep stable context stable

This changed how I review tool and prompt composition. Tool metadata is not free scaffolding. It is part of every model request, contributes to startup context, and can decide whether large stable sections remain cacheable. Any session-specific data inserted near the beginning can make every following token cold, even if the agent never uses the tool that carries it.

I now ask four questions when adding a dynamic value to an agent harness:

1. Which client actually reads it?
2. Can it be sent in a later message or private channel instead of the shared tool schema?
3. How early does it appear in the serialized request?
4. Can two otherwise identical sessions produce byte-identical prefixes before the first truly user-specific input?

The answer is often to split context by stability: fixed tool definitions and shared policy first; run-specific routing and scratch metadata later; secrets never in model-visible text. That structure makes cache reuse a property of the protocol, not a guess based on token billing.

One boundary remains: fixing the tool-description marker does not remove the separate session-specific path in the first message. I treat it as an independent source of prefix divergence, not as a fix already delivered by this change. The marker change was pushed with a test but still awaited release at the end of the verified source window.

Put changing data after the reusable prefix

When I inspect a cache miss, I look for the first differing byte in the serialized request, not just the first difference in the visible prompt. Tool descriptions, injected routing metadata and system additions can all precede user content. If a unique session marker is attached to an early tool definition, moving only a path in the user message will not restore reuse. The stable prompt has to stay stable before the variable marker appears.

The design constraint is ordering. Put shared instructions, stable tools and reusable examples first. Put per-session identifiers, transient paths, recent state and task-specific data later, preferably in a compact structured field. Do not remove a value that the runtime needs for routing or audit; move it to the latest safe point in the request. Then compare actual serialized bodies across fresh sessions and inspect cache-read/cache-write events separately.

Place dynamic data after the reusable prefix

The cache boundary follows serialization order. If a unique value appears near the top of the request, the shared suffix after it may no longer match, even when the unique value itself is tiny. That makes a one-line change capable of invalidating a much larger reusable prefix. Tool definitions are part of that serialized prompt, so descriptions deserve the same review as system instructions and examples.

My practical check is to capture two requests that should share a prefix, normalize only values that are intentionally variable, and locate the first byte where they differ. Then I compare the provider’s cache accounting, because a textual diff tells me where uniqueness starts but not how much of the request was actually reusable. The test needs a negative control: put the unique line back into the shared region and verify the common-prefix measure drops. Otherwise a parser bug or an incorrect request capture could make the experiment look successful.

The fix is not to delete every dynamic value. Some clients need a session-specific binding to call tools correctly. I put that value at the narrowest client-specific boundary that consumes it, and keep the shared tool schema stable for the other clients. I also keep the separate scratch-path difference visible rather than claiming it was solved by this change. That distinction protects the cache result from turning into a broader claim about all per-session prompt variation.

Test the actual serialized request

The unit under test is the request the provider receives, after the harness has assembled tool schemas and session data. A local object model can look stable while a later serialization step inserts a per-session value ahead of the common prefix. I therefore compare captured request bytes from the real client path and make the test fail if a supposedly stable field changes by session. I also keep the cache accounting alongside the capture, because only the provider-side observation tells us whether the expected bytes were reused. This prevents a prompt-template test from being mistaken for a prompt-cache result.

A good experiment changes one field at a time. Keep the same model, tool set and task; vary only the session marker; then confirm that the expected prefix length becomes reusable. A prompt diff shows where bytes differ, but meter data shows whether the provider treated the shared prefix as a cache hit. Have you found the earliest session-specific value in your own serialized request?

How this was verified

I checked the captured-request analysis and the exact server/test change at its pinned commit. The cache boundary explanation follows the serialized prefix order; the article does not claim realized savings or that the separate scratch-path difference was resolved.