Measure accepted work before changing the hiring bar.

Video caption

Measure accepted work before changing the hiring bar.

Selection is a first-order measurement problem.

Attention and release gates may drive the result.

My rule: Keep fundamentals and decision quality in view.

#EricFieldNotes

Full written post / accessibility read

If we hire engineers for agent orchestration, we should be able to show that the organization delivers more accepted product value. Count the full work from request through review, QA, deployment and customer readback. First-diff speed is useful telemetry, but it is not the team outcome.

Compare similar tasks within the same team and predefine acceptance. Log work declined by agents, abandoned attempts, multiagent overlap, reviewer time and post-release correction. An average across easy and hard work can flip merely because the mix changed. METR's 2026 update flags selection and timing limitations explicitly.

If the team adds bounded agent lanes, independent QA and decision packets, compare accepted-change throughput, defect burden and owner review time before and after. Track product mix and concurrent organizational changes. DORA's amplifier framing is a warning that the surrounding system matters; it does not prove this package caused an uplift.

My rule: optimize cost and time per accepted outcome, with rework and incident cost visible. Do this because agents shift where effort occurs, and hiring should target the human decisions that actually improve the full path. Validate that claim against your own work rather than the first appealing chart.

#EricFieldNotes

Four-beat scene transcript

1. Measure accepted work before changing the hiring bar.

If we hire engineers for agent orchestration, we should be able to show that the organization delivers more accepted product value. Count the full work from request through review, QA, deployment and customer readback. First-diff speed is useful telemetry, but it is not the team outcome.

Visual: A faster patch is not necessarily a faster team.

2. Stratify and keep the misses.

Compare similar tasks within the same team and predefine acceptance. Log work declined by agents, abandoned attempts, multiagent overlap, reviewer time and post-release correction. An average across easy and hard work can flip merely because the mix changed. METR's 2026 update flags selection and timing limitations explicitly.

Visual: Selection is a first-order measurement problem.

3. Test the operating change, not the slogan.

If the team adds bounded agent lanes, independent QA and decision packets, compare accepted-change throughput, defect burden and owner review time before and after. Track product mix and concurrent organizational changes. DORA's amplifier framing is a warning that the surrounding system matters; it does not prove this package caused an uplift.

Visual: Attention and release gates may drive the result.

4. Use the result to redesign work and hiring.

My rule: optimize cost and time per accepted outcome, with rework and incident cost visible. Do this because agents shift where effort occurs, and hiring should target the human decisions that actually improve the full path. Validate that claim against your own work rather than the first appealing chart.

Visual: Keep fundamentals and decision quality in view.

Research and claim limits

• DORA State of AI-assisted Software Development 2025 (S152)
• METR early-2025 experienced-developer RCT (S153)
• Vella and Blincoe, longitudinal AI coding-assistant study (S154)

The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.