FIELD NOTE / X
A benchmark win is not your release gate.
The complete written thought and the evidence behind it. The video edition will follow its public release.
The written argument is here.
This approved X edition is on the journal now. Its video player and original platform link will appear after each public release is verified.
A benchmark win is not your release gate.
Video caption
A benchmark win is not your release gate. Freshness and grading change the score. Use local evidence to authorize action. #EricFieldNotes
Full written post / accessibility read
A coding agent can perform well on repository bug benchmarks while still failing your product's actual acceptance contract. Benchmark tasks have their own repo mix, issue wording and test design. Your team has current code, customer exceptions, hidden dependencies and a release decision that the leaderboard never measured.
SWE-bench Live reports a gap from static benchmark performance on fresher, broader issues. SWE-Bench Pro Verified documents leakage and flawed-task concerns that can inflate results. Neither paper tells you which agent will satisfy a particular tenant entitlement rule in your codebase.
Select representative current tasks and write the product decision, affected cohort, forbidden behavior and expected readback before letting the agent work. Pin model and harness versions, isolate the worker, and have an independent grader run on the exact patch. Include a deliberate failing mutant to test the test.
My rule: let a public benchmark narrow the search, then judge agents on accepted outcomes in your own task classes with quality, delay, human rescue and cost included. Do this because a score for someone else's issues cannot certify your customer's promise.
#EricFieldNotes
Four-beat scene transcript
1. A benchmark win is not your release gate.
A coding agent can perform well on repository bug benchmarks while still failing your product's actual acceptance contract. Benchmark tasks have their own repo mix, issue wording and test design. Your team has current code, customer exceptions, hidden dependencies and a release decision that the leaderboard never measured.
Visual: The task and oracle may be different.
2. The papers show why caution matters.
SWE-bench Live reports a gap from static benchmark performance on fresher, broader issues. SWE-Bench Pro Verified documents leakage and flawed-task concerns that can inflate results. Neither paper tells you which agent will satisfy a particular tenant entitlement rule in your codebase.
Visual: Freshness and grading change the score.
3. Run a local acceptance slice.
Select representative current tasks and write the product decision, affected cohort, forbidden behavior and expected readback before letting the agent work. Pin model and harness versions, isolate the worker, and have an independent grader run on the exact patch. Include a deliberate failing mutant to test the test.
Visual: Keep the expected outcome outside the worker's tree.
4. Use benchmarks to choose candidates.
My rule: let a public benchmark narrow the search, then judge agents on accepted outcomes in your own task classes with quality, delay, human rescue and cost included. Do this because a score for someone else's issues cannot certify your customer's promise.
Visual: Use local evidence to authorize action.
Research and claim limits
- SWE-bench Live paper (S22)
- SWE-bench Pro Verified paper (S23)
The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.