A model swap is an architecture decision.

Video caption

A model swap is an architecture decision.

Otherwise the leaderboard is a category error.

p95, review, rework and wrong-branch cost.

My rule: An owner approves the accepted-work frontier.

#EricFieldNotes

Full written post / accessibility read

Before replacing a small generative route with a decision model, draw the complete path. Does the new route need a writer afterward? More state preparation? A review queue? A different failure and retry contract? Single-call latency cannot answer that.

A Jev Choice, Haiku or Qwen response, deterministic rule and human escalation often produce different artifacts. Define the user-visible accepted answer and the exact policy outcome first. Then give each candidate the same source facts and protected labels.

Measure warm and cold latency, network, generation, retries, reviewer minutes, abstention and re-open rate by case slice. If one route is locally hosted, include concurrency, hardware cost and recovery. Record the point where a person takes over.

Choose the smallest path that meets the outcome, tail-latency and recovery budget, and re-evaluate after model or policy changes. Do this because a model swap changes the whole service contract, even when the API type stays the same.

#EricFieldNotes

Four-beat scene transcript

1. A model swap is an architecture decision.

Before replacing a small generative route with a decision model, draw the complete path. Does the new route need a writer afterward? More state preparation? A review queue? A different failure and retry contract? Single-call latency cannot answer that.

Visual: The fastest call can produce the slowest accepted job.

2. Compare on the same job.

A Jev Choice, Haiku or Qwen response, deterministic rule and human escalation often produce different artifacts. Define the user-visible accepted answer and the exact policy outcome first. Then give each candidate the same source facts and protected labels.

Visual: Otherwise the leaderboard is a category error.

3. Price errors and recovery.

Measure warm and cold latency, network, generation, retries, reviewer minutes, abstention and re-open rate by case slice. If one route is locally hosted, include concurrency, hardware cost and recovery. Record the point where a person takes over.

Visual: p95, review, rework and wrong-branch cost.

4. Buy the route, not the model story.

Choose the smallest path that meets the outcome, tail-latency and recovery budget, and re-evaluate after model or policy changes. Do this because a model swap changes the whole service contract, even when the API type stays the same.

Visual: An owner approves the accepted-work frontier.

Research and claim limits

• TypeSafe AI: Introducing System One Models and Jev (S97)
• TypeSafe AI: Jev 1.13 jaggedness (S99)
• Li et al., JEV-as-a-Judge (S103)
• OpenAI: Structured Outputs guide (S175)

The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.