Make model failover preserve the agent contract

A fallback can produce a perfectly readable answer while violating the conditions that made the original route acceptable.

I reproduced that distinction with our runner using scripted candidates. A synthetic rate failure moved to the next candidate, as intended. A synthetic timeout did too. A synthetic Gemini safety block became an empty-result failure and also moved to the next candidate. No real provider was contacted; the experiment exposed the runner's decision, rather than measuring a model.

The architecture question is larger than retrying a request. A route change can alter tool representation, role separation, privacy, region, cost accounting and the meaning of a refusal. I want availability within the agent's contract. That requires defining what must survive a switch before choosing the next available model.

Treat the fallback as another execution route

The runner has useful availability mechanisms already. It rebuilds candidates, orders them by tier and recent health, applies cooldowns, respects a shared deadline and rejects unusable output. Pool routes precede service fallbacks. The code also checks an owner model restriction before calling a candidate and again against the model reported in the returned answer.

Those are concrete controls, but they do not prove feature parity. A candidate can be healthy and inexpensive while representing the same conversation differently. A route that supports ordinary text may not support write tools, vision, the required region or a particular privacy condition. Successful transport is only one prerequisite for an acceptable answer.

I therefore separate route preference from route eligibility. Preference can optimize cost, recent failure rate or latency among acceptable candidates. Eligibility decides whether the route may receive this turn at all. Mixing the two lets a ranking algorithm accidentally become a policy exception mechanism.

The route record also needs ownership and provenance. A capability declaration is a claim about an adapter, configuration and provider endpoint. It needs a qualifying replay and a validity boundary. A mutable boolean written by the same worker that wants permission is not an authenticated guarantee.

Classify the failure and the dispatch state separately

A rate limit before execution, a provider safety refusal and a timeout after dispatch are different states. Calling all three retryable errors hides the most important information.

For a known pre-execution rejection, another eligible route may be appropriate. For a safety refusal, switching providers to obtain a different answer changes the policy being enforced. For unknown dispatch, the system first needs reconciliation: did the previous attempt execute, bill, emit a tool proposal or complete an external operation?

The QAC design record explicitly calls for this separation. The pinned runner still moves to the next candidate after every caught failure. My scripted probe confirms that behavior, including a safety block classified as empty output. The proposed policy below is a new teaching model of the desired contract, not a claim that this replacement is activated in the gateway.

def disposition(failure: str, dispatch: str) -> str:
    if failure == 'safety_refusal':
        return 'terminal_refusal'
    if dispatch not in {'not_sent', 'rejected_before_execution'}:
        return 'reconcile_unknown'
    if failure in {'rate', 'unavailable', 'quota'}:
        return 'consider_route'
    return 'hold'

The ordering is deliberate. A refusal remains terminal even if another field suggests the request was not dispatched. Unknown dispatch remains conservative even when the failure name sounds retryable. Unrecognized failure classes hold rather than quietly inheriting a broad retry default.

Keep model attempts separate from tool effects

A failed model request and a failed tool operation are not interchangeable. The model may have returned a proposal that the orchestrator has not executed. Alternatively, an effectful tool request may have reached a remote service before the connection failed. The retry policy needs the actual operation lifecycle, not a guess based on an HTTP timeout.

A new provider route should retain the same logical operation identity. That identity connects attempts, receipts, approvals and any reconciliation action. It does not itself prevent duplication. The effectful service still needs an idempotency or receipt contract, and an owner must decide what to do when the service cannot establish an outcome.

I would not let a fallback recreate an approval merely by regenerating equivalent-looking arguments. Approval belongs to the authorized payload and context. If the new route proposes a different consequential action, the old approval cannot silently expand to cover it. Provider availability is not a grant of additional authority.

This keeps the boundaries understandable. The inference router selects an eligible model attempt. The tool broker validates current authority. The target service owns its effect receipt. A route switch must preserve those relationships rather than treating a fresh model answer as a fresh transaction.

Build a capability matrix around the turn

The turn has requirements; the route has qualified capabilities. The eligibility check compares the two before any prompt leaves the process.

My proposed exercise represents requirements for tools, write tools, untrusted content, vision, zero retention, region, quality, deadline and remaining budget. A write-bearing turn requires a native write-capable route in this deliberately narrow model. An emulated read-only route is ineligible, even if it can produce convincing text about the requested write.

The distinction between tools being offered and tools being used matters. A turn may need a safe representation of tool authority before the model has chosen to invoke anything. Waiting until after generation to discover that the fallback flattened tool results into user text is too late to preserve the original boundary.

For this model I chose a conservative rule: all routes must separate instruction roles, and turns using tools or previously exposed to untrusted content also require honest tool representation. This is an explicit teaching policy. A production capability matrix needs adapter-specific qualification and cannot rely on these trusted synthetic declarations alone.

Preserve role separation through the adapter

The original to_chat adapter keeps instructions in a system message and function results in tool messages. I tested that with a synthetic tool result containing a forged user-style label. In that fixture, the label remains content of a tool message; it does not become a new user instruction.

That successful fixture does not qualify every route. The design record separately identifies relay flattening and misleading read-only tool descriptions as parity work. A common outer response shape cannot prove the inner request preserved its trust boundaries. The acceptance test must inspect the actual adapter output.

Role separation is necessary, but a test should not pretend it makes prompt injection impossible. The system still needs scoped tool authority, data boundaries and service enforcement. The adapter's obligation is narrower: preserve which content came from which authority and describe the available tools accurately.

I use the same adversarial fixture across routes. Instructions, a write tool and a forged role label in a tool result create a useful comparison. Each route must show where those elements land. If a route uses emulation, its fencing and escaping rules need their own negative controls.

Privacy is an eligibility condition

The pinned OpenRouter request body carries three privacy-routing fields: deny data collection, require zero-retention endpoints and require support for the sent parameters. The local adapter fixture verifies those fields remain in the generated body.

That is evidence about our request construction. It is not a new audit of provider retention, a promise about every endpoint or proof of where the request was processed. Those are separate qualification obligations. An endpoint without the required behavior should make the route unavailable rather than cause the client to remove the privacy requirement.

The design also calls for organization-level router restrictions and a capability matrix with region and credential kind. A customer contract that excludes a third-party router must survive every exhausted pool and every fallback hop. Availability should degrade to a clear hold if no eligible route remains.

A privacy requirement cannot be merely another ranking weight. A cheaper route does not compensate for being forbidden. I would place privacy and region checks before health-based ordering, and preserve the rejection reason in an operational receipt that contains no private prompt or tool output.

Unknown price cannot win as zero cost

The source has a concrete accounting trap: its Gemini price table contains a different model from the configured default route. When the adapter cannot find a price, it uses zero. My offline fixture supplies one thousand input tokens and ten output tokens and observes a reported cost of zero for that missing-price route.

Those token counts are synthetic inputs, not a usage measurement. The result demonstrates missing-price behavior. It does not establish free inference, a bill or current model pricing. The useful correction is conceptual before it is arithmetic: unknown price must have its own state.

The proposed selector requires a known nonnegative reservation in integer micros. A real zero reservation is representable; a missing reservation is rejected. The budget must cover the candidate's reservation before dispatch. That local comparison is not a transactional reservation service and does not prevent concurrent workers from spending the same remaining balance.

A production implementation needs one usage leg per attempted hop, including unsuccessful attempts where usage exists. Subscription allocation, list-equivalent estimates and billed usage should remain separate. The final answer's model cannot stand in for the cost of every attempt that preceded it.

Spend one deadline across the fallback sequence

The current runner computes remaining time against a shared deadline before each candidate. That is a useful invariant: a route switch does not create an entirely new conversation budget. The teaching model makes the boundary explicit by comparing remaining time with a candidate's declared minimum.

At the exact required remainder, the candidate is eligible. One millisecond below it, the candidate is refused. Boundary tests matter because an off-by-one policy can either dispatch work with insufficient capacity or unnecessarily reject an otherwise valid final attempt.

The minimum is still a declaration. It is not a latency guarantee. Real network timeouts, cancellation behavior, queueing and provider work already in flight need their own accounting. The system should retain a receipt for an attempt that outlives the caller's wait, rather than infer that the work disappeared when the client stopped listening.

Retry backoff belongs inside the same deadline and budget. The QAC plan calls for full jitter; the proposed selector here does not implement scheduling or jitter. Keeping that omission explicit prevents a small eligibility function from being mistaken for a complete resilient inference service.

Preserve refusals without turning them into incidents

A terminal refusal should return a clear typed outcome to the caller. It need not become an infinite critical alert, an unowned queue item or a prompt to try a different model secretly. The caller needs to know the step that refused, what did and did not execute, and which owner can resolve any legitimate policy question.

The same applies to no eligible route. A capability mismatch is different from exhausted budget, expired deadline or unavailable provider. Operators need those reasons to repair capacity or configuration without weakening the original contract. A generic all-failed string discards the distinction.

This is an availability design problem. Teams often optimize for always returning something because visible silence looks bad. Returning an answer through an ineligible route can be harder to detect and more consequential. A well-designed hold gives the product a useful explanation and a bounded next action without inventing success.

I would keep the public response concise and the operational receipt specific. The user should not need to understand provider internals to know the task is held. The owner should be able to identify the failed eligibility condition without reading private conversation content in logs.

Test route policy separately from provider quality

The proposed policy exercise uses synthetic route records and deterministic transitions. It checks terminal refusal, unknown dispatch, eligible retry classes, deadline and budget boundaries, role and tool requirements, region, router permission, privacy, quality and attempted-route exclusion. It also checks that the first eligible route is selected and that an empty eligible set produces a hold.

When no eligible candidate exists, the model returns hold. It does not call a provider and hope the response is acceptable. An eligible route can still fail later, which produces another attempt record and a new classification decision.

The exercise cannot measure model judgment, answer quality or provider latency. Its quality threshold is a chosen trusted score, not a benchmark result. A fixed replay corpus is the next qualification layer: same conversations, same adversarial tool outputs, known expected outcomes and measured intervention, usage and latency per route.

I keep that distinction in the source packet. A policy can be correct over its declared inputs while the capability registry is wrong. A provider can pass a happy replay while failing a refusal or tool-boundary case. The acceptance system needs both, plus the independent mutation controls that show each detector can actually fail.

Keep the audit leg useful without retaining the prompt

The current runner logs candidate identity, tier, failure kind, elapsed milliseconds and cooldown. Those fields can explain a route sequence without copying the conversation into operational telemetry. They also need precise interpretation: an elapsed hop includes wrapper and adapter work, not isolated model inference time.

I would link each leg to the stable operation identity and exact route configuration. Record the eligibility decision, dispatch knowledge, usage availability and terminal outcome. A missing receipt should remain missing, rather than becoming an inferred zero-cost failure. The authorized reviewer can then reconstruct why a route was selected and whether the next action respected the contract.

This audit structure supports incident work and architecture decisions. If one fallback repeatedly needs human reconciliation, that burden belongs in its operating cost. If a route becomes unavailable whenever strict privacy is required, capacity planning needs to account for that reduced pool. Neither conclusion requires retaining private tool output in a general logging system.

The valuable comparison is the decision sequence and its accepted outcome. Keep the private evidence in its authorized system, and give operational telemetry the minimum references necessary to locate it.

Give the fallback a release contract

I would ship a new route only with a versioned capability record, adapter replay, known accounting behavior and a clear owner. The release should identify the exact model and harness configuration, permitted data classes, processing constraints, tool representation and evidence expiry. If any of those change, the route needs requalification rather than inheriting yesterday's approval by name.

The comparison should include the entire accepted outcome: failed attempts, review, human intervention and reconciliation. Fast first-token output is useful, but it is not the endpoint when a task must preserve authority across tools and services. A route that answers quickly and causes a later manual repair may be more expensive than a slower eligible route.

The operator question is simple enough to use in a design review: what conditions made the primary route acceptable, and where do we prove those same conditions before dispatching to the fallback? The answer should be an executable contract with a clear hold state, not a list of provider names.

How this was verified

The packet pins the full runner source and QAC retry/parity design at an immutable Git commit. Current behavior was reproduced with scripted candidates and adapter fixtures, without credentials, external inference or tool execution. The original local-server owning tests are recorded separately from the proposed policy exercise and independent review.

The proposed selector is new educational code over trusted declarations. It is not installed in the runner, does not authenticate capability records, reserve budget transactionally, execute tools or prove current provider parity. Synthetic token inputs demonstrate missing-price behavior only. No current pricing, provider quality comparison or new production activation is claimed.