A fail-closed allow-list can silently disable your failover

Here is the pattern to watch for. You tighten an input check on a model relay for good safety reasons. The primary backend keeps answering every request, so nothing looks different. Meanwhile the new check refuses every request that reaches the backup path, because the real client never sends what the check expects. Your failover is now switched off, and the success numbers cannot tell you. The day the primary has a bad day, you find out all at once.

This shape can run for about two weeks before anyone notices. That is long enough for a backup to become a story you tell yourself.

What the setup looks like

The example is a small relay that sits between an internal app and the models. It speaks one common model API format to the app. Behind it is a pool of slots. It tries Codex slots first. If none of them can answer, it falls through to Claude slots, which run the Claude CLI non-interactively with tools disabled. The client never needs to know which backend answered. That is the whole point of a pool: one provider can have a bad day and the app keeps working.

The client asks for a Codex model id, gpt-6.1-sol. When a Claude slot answers, the response's model field comes from the CLI's own usage report, so the client sees the model that actually answered rather than the one it asked for.

Before the problem, the relay handled about eleven requests a day, and nearly all of them worked.

How a safety change turns off the backup

The change was a reasonable one. The Claude call path was tightened so it would accept only two exact model ids, claude-sonnet-5-5 and claude-opus-5-5. Anything else raised an error before the command line was even built.

On its own terms, that is a good rule. A caller cannot slip an arbitrary string into the arguments of a CLI call, and the relay cannot quietly serve a model nobody approved. It fails closed, which is what I recommend.

The catch was the one real client. It never sends a Claude id. It sends gpt-6.1-sol. From the day of the change, every request that reached a Claude slot was refused with an "unsupported exact Claude model" error. The failover path was dead.

Nobody saw it, because nothing reached it. The Codex slots answered every request, so the refusal never showed up in the success numbers. For about two weeks the relay looked healthy, and in a narrow sense it was. It just had no backup.

Then the Codex slots stopped working. The slot error said the access token had expired. The underlying cause was that the Codex account had hit its usage limit for several more days, and getting it back needed a manual re-login. Every request fell through to the Claude slots, and every one was refused on the model id. One day showed nine requests with no successes. The next morning showed three more, also with no successes. Twelve of twelve failed over about a day and a half.

So the failure was not "a provider went down". The failover had been switched off two weeks earlier by a change made for safety, and the provider outage only exposed it.

No alarm fired at any point, before or during the outage. Nothing paged anyone.

Why it happens

Three things line up.

Fail closed is a property of the code. Failover is a property of the system. You can write a perfect refusal and still never test that the backup path accepts what real traffic sends. The allow-list was the fail-closed half done well. The failover half was never exercised with the real client's payload.

Allow-lists get written from what callers ought to send. The relay's own log had the client's model id on every line. A look at the distinct values in that log before the change would have shown gpt-6.1-sol sitting outside the list.

A refusal on a fallback path looks like a normal error, and zero traffic looks like a quiet day. Nothing marked a backup-path refusal as special. Nothing watched the success count. So a correct safety decision became an outage that looked like inactivity.

How to detect it

The checks that would have caught this are small.

• Force the backup path with real payloads. One realistic request carrying the real client's model id, with the primary deliberately skipped, would have failed on day one instead of day fifteen.
• Alert on zero successful requests over a window where you normally expect traffic. Keep that separate from error-rate alerts. A stretch with no successes is an alert, not a quiet day.
• Log requested versus served model on every response. Then substitutions and refusals are visible without reading code.
• Count refusals per path. A refusal on the primary path and a refusal on the backup path mean different things. Give the backup path its own counter that someone sees.

A related published note is Make model failover preserve the agent contract. This piece is about an earlier question: whether the failover exists at all.

How to prevent it

The repair kept the safety rule and taught it about the one caller that actually exists.

1. Map the real client's id instead of refusing it. A new step checks whether the request carries a bare id from the client's model family, using a strict full match on that family's id format. If it does, the request runs on the configured Claude model pin, claude-sonnet-5-5. Every other value still goes through the original exact allow-list, and anything not on it is still refused before the arguments are built. The fail-closed rule stays.
2. Add an operator switch to skip the primary. A setting lets an operator skip the Codex slots until the account works again. It mirrors an existing switch for skipping the Claude slots. It also makes scheduled backup probes possible, because you can force traffic onto the backup on purpose.
3. Log the substitution. The success log line records a sanitized label of the requested model: a short run of plain letters, digits, dots, underscores and hyphens, and invalid for anything else. When a gpt-6.1-sol request is served by Claude, the log says so.
4. Keep the response truthful. The response's model field still comes from the CLI's usage report. The client sees claude-sonnet-5-5, never an invented gpt-6.1-sol.

Re-logging into the Codex account was left out of the repair on purpose. The account was at its usage limit for several more days anyway, and the point was to make the backup carry the load.

Prove the tests can fail

The part I care about most is the failing control. A test suite written alongside a fix proves very little if it would also pass on the broken code. So the new tests were also run against the relay as it was before the change.

The three failures against the old code were the two failover cases and the switch test, which are exactly the behaviors the fix was meant to add. The 37 tests are the whole test file for the component. There were no other relay tests, which is its own finding. A relay that guards a real user path should not depend on a single test file, and certainly not on one with no test that sends the real client's id.

An independent red-team review of the first version passed the code with follow-ups. Those went in before the final version: the requested-model log label, refusal cases for ids carrying a newline, fullwidth characters or a trailing newline, a stale comment and a manual note. Then a live in-process smoke test on the new host sent a gpt-6.1-sol request through a real Claude slot. It returned 200, with model claude-sonnet-5-5 and the Claude backend.

The general version of that habit is in A merge gate must prove the tests ran: a pass count means little until you also know the failure count on the old code.

Keep emergency fixes short-lived

When a service is fully down, the first repair often lands on the live host before it is committed. That can be the right call for a dead service, but it is a state to keep short. Record the decision in a durable log when you make it. Open a reviewed change that regularizes the live state. Write the rollback down before you need it: here, revert the change, remove the skip-primary setting and restart the service.

Fail closed needs a health signal

The rule I take from this extends what I usually say about gateways. I recommend one gateway in front of your models that fails closed and has tested failover. Those two halves can work against each other. A strict refusal on the fallback path, with no alarm, is a silent outage waiting for its day.

So every fail-closed decision on a fallback path should count against a health signal someone sees. Refusing a bad request is correct. Refusing every request on the backup without anyone knowing is not a safety property. It is an outage with good intentions.

For the design side of this, where refusals and recovery states are part of the state machine rather than afterthoughts, see Design recovery semantics into the safety state machine.

Checklist

Copy this into the review template for any change that touches a relay, a gateway or a fallback path.

[ ] Before tightening an allow-list, pull the distinct values seen in
    the logs and confirm the new list accepts every legitimate one.
[ ] Force the fallback path with the real client's payload, primary
    skipped, after every change to either path.
[ ] Schedule that fallback probe; do not rely on a human remembering.
[ ] Alert on zero successful requests over a window that normally
    has traffic, separately from error-rate alerts.
[ ] Count refusals on the fallback path separately and surface them.
[ ] Log requested model and served model on every response.
[ ] Keep the response truthful about which model answered.
[ ] Run new tests against the previous version and record the failure
    count next to the pass count.
[ ] Treat "this component has one test file" as a gap to close.
[ ] If an emergency fix lands live before commit, log the decision,
    open a reviewed change and write the rollback the same day.

The probe is the item I would do first. Until a scheduled request forces the backup path with real traffic, the failover is a claim, not a property.

Go deeper: the Transition2 course covers gateways, fail-closed design and the checks that keep a backup real.