FIELD NOTE / LINKEDIN
Protect your local eval from benchmark theater.
The complete written thought and the evidence behind it. The video edition will follow its public release.
The written argument is here.
This approved LinkedIn edition is on the journal now. Its video player and original platform link will appear after each public release is verified.
Protect your local eval from benchmark theater.
Video caption
Protect your local eval from benchmark theater.
Use classes, not one cherry-picked prompt.
A deliberate wrong patch should fail.
My rule: Both need evidence at their own boundary.
#EricFieldNotes
Full written post / accessibility read
Fresh coding benchmarks are useful, but I would not equate their score with accepted work in a company's current repository. SWE-bench Live highlights static-set limitations. SWE-Bench Pro Verified shows how leakage and task defects can distort grading. Neither knows your customer's contract or your release boundary.
Include recent bug fixes, cross-service changes, migration decisions and browser-facing behavior. Prewrite source-linked acceptance outside the worker's editable tree. Pin model, harness, tools, task snapshot and test budget. Record all attempts, human rescue and cost per accepted result, not only the best patch.
Inject a disposable mutant that violates a protected entitlement rule while preserving a plausible UI path. If the grader stays green, the score is not supporting the claimed product behavior. Bind the passing result to the exact commit and build digest so yesterday's report cannot bless today's artifact.
Use external benchmarks to shortlist candidates and local accepted-task data to choose a model or route. Use a separately trusted release gate to authorize a particular build. Do this because even a valid model selection experiment cannot substitute for evidence that this specific change preserves the product decision.
#EricFieldNotes
Four-beat scene transcript
1. Protect your local eval from benchmark theater.
Fresh coding benchmarks are useful, but I would not equate their score with accepted work in a company's current repository. SWE-bench Live highlights static-set limitations. SWE-Bench Pro Verified shows how leakage and task defects can distort grading. Neither knows your customer's contract or your release boundary.
Visual: A public score answers a narrower question.
2. Build a local task matrix.
Include recent bug fixes, cross-service changes, migration decisions and browser-facing behavior. Prewrite source-linked acceptance outside the worker's editable tree. Pin model, harness, tools, task snapshot and test budget. Record all attempts, human rescue and cost per accepted result, not only the best patch.
Visual: Use classes, not one cherry-picked prompt.
3. Test the evaluator itself.
Inject a disposable mutant that violates a protected entitlement rule while preserving a plausible UI path. If the grader stays green, the score is not supporting the claimed product behavior. Bind the passing result to the exact commit and build digest so yesterday's report cannot bless today's artifact.
Visual: A deliberate wrong patch should fail.
4. Keep selection and authorization separate.
Use external benchmarks to shortlist candidates and local accepted-task data to choose a model or route. Use a separately trusted release gate to authorize a particular build. Do this because even a valid model selection experiment cannot substitute for evidence that this specific change preserves the product decision.
Visual: Both need evidence at their own boundary.
Research and claim limits
- SWE-bench Live paper (S22)
- SWE-bench Pro Verified paper (S23)
- GitHub Docs: ruleset required status checks (S146)
The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.