A model leaderboard cannot buy your workflow.

Video caption

A model leaderboard cannot buy your workflow.

A mistaken escalation and a mistaken approval cost differently.

Rule, typed judge, small generator and human fallback.

My rule: The answer changes when the case mix or model changes.

#EricFieldNotes

Full written post / accessibility read

A decision model, a small hosted writer and a local open-weight model make different promises. The interesting procurement unit is a full service route that produces an accepted result at a known cost, latency and risk. One call's benchmark cannot supply that frontier.

In an illustrative operations queue, routing one ordinary case to review wastes minutes. Auto-approving one restricted case can trigger days of repair. A single accuracy percentage conceals that asymmetry. The threshold belongs to the business owner, using case slices and explicit consequence weights.

Run the same protected cases through each complete route. Measure accepted volume, severe wrong branches, p ninety-five completion, reviewer minutes, reopens and marginal cost under burst load. Include an abstain path and mark what the system cannot safely decide.

Let an owner approve the cheapest route that satisfies each slice's outcome and recovery budget. Re-test after policy, model or load changes. Do this because the economical architecture is conditional on the work you actually accept.

#EricFieldNotes

Four-beat scene transcript

1. A model leaderboard cannot buy your workflow.

A decision model, a small hosted writer and a local open-weight model make different promises. The interesting procurement unit is a full service route that produces an accepted result at a known cost, latency and risk. One call's benchmark cannot supply that frontier.

Visual: Procurement must price validation and handoff.

2. The error bill is asymmetric.

In an illustrative operations queue, routing one ordinary case to review wastes minutes. Auto-approving one restricted case can trigger days of repair. A single accuracy percentage conceals that asymmetry. The threshold belongs to the business owner, using case slices and explicit consequence weights.

Visual: A mistaken escalation and a mistaken approval cost differently.

3. Test a route portfolio.

Run the same protected cases through each complete route. Measure accepted volume, severe wrong branches, p ninety-five completion, reviewer minutes, reopens and marginal cost under burst load. Include an abstain path and mark what the system cannot safely decide.

Visual: Rule, typed judge, small generator and human fallback.

4. Choose a frontier, then monitor drift.

Let an owner approve the cheapest route that satisfies each slice's outcome and recovery budget. Re-test after policy, model or load changes. Do this because the economical architecture is conditional on the work you actually accept.

Visual: The answer changes when the case mix or model changes.

Research and claim limits

• TypeSafe AI: Introducing System One Models and Jev (S97)
• TypeSafe AI: Jev 1.13 jaggedness (S99)
• Anthropic: Models overview (S104)
• Qwen: Qwen3 model family (S105)
• TypeSafe AI: Jev interface (S176)

The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.