Decompose work around contracts and integration costs

The Sonnet whole-task build recorded 60 of 60 hidden checks in 35.6 minutes. The serial-pieces build recorded 39 of 60 in 92.3 minutes, despite passing 1,025 of its own tests.

That result changed the question I ask before dividing work among agents. The question is where independently owned work can still compose into a correct system, within the time available for integration and acceptance.

Our historical experiment used one production-style webhook-delivery service, a shared task and template, common checks and a nominal 90-minute cap. The pieces condition was a sequence of fourteen assignments with a check after each. It was not fourteen agents working concurrently.

Four models had completed arms under both methods. In those four matched pairs, the whole-task arm recorded higher hidden completeness and less builder wall time. The cost records also favored whole-task within each pair, with an important accounting distinction: three pairs used subscription list-price equivalents; the Gemini pair used billed usage.

The useful lesson is about contracts, state ownership and the integration budget. A method can improve local test attention while leaving the complete delivery system unfinished.

Preserve the experiment you actually ran

The task was a Relay webhook-delivery service with PostgreSQL, multi-tenant authorization, idempotency, a delivery worker, retry behavior, rate limiting, metrics and shutdown handling. Those requirements interact. A delivery admitted under an idempotency key must fan out under the correct tenant, survive worker interruption and preserve its retry and completion semantics.

The whole condition gave one builder the coherent task. The pieces condition provided fourteen serial assignments with local checks. Every additional piece could inspect the repository and make progress against its local target, but the final system still had to satisfy the original contract.

I keep that distinction explicit because the result is easy to misstate as evidence against parallelism. We did not compare a whole-task agent with a well-designed concurrent team sharing stable interfaces and an integration owner. The observed treatment combined serial decomposition, repeated setup and repeated checking under a fixed deadline.

The next experiment should vary those factors deliberately. Changing concurrency, acceptance criteria, harness and model simultaneously would make another attractive table difficult to interpret. A decomposition method needs a precise description before its result can guide an operating decision.

Compare completed pairs before comparing models

The recovered metric receipts support these four within-model pairs:

These are recorded builder durations and hidden-suite results. Wall time includes harness and tool work; it is not isolated inference latency. The nominal cap and observed elapsed time are separate values, which is why some recorded times exceed ninety minutes.

The Flash whole selection is the scored third run. The retained report identifies the earlier repetitions separately. Selecting the first matching directory name or averaging every attempted run would produce a different population from the comparison in the report.

The reanalysis validates recorded hidden-score structures before comparing pairs:

def _score(metric):
 value=metric.get('hid')
 if not isinstance(value,list)or len(value)!=3:
  raise ValueError('missing hidden score')
 passed,total,failed=value
 if type(passed)is not int or type(total)is not int or total<=0 or passed<0 or passed>total or not isinstance(failed,list)or len(failed)!=total-passed:
  raise ValueError('invalid hidden score')
 return passed,total

The accompanying analysis fixes those selections explicitly. It refuses duplicate arm identities, missing selected arms, model or effort mismatches and different hidden-suite denominators. It validates the recorded score and numeric fields before computing ratios. That makes the analysis reproducible from the recovered data without pretending to recreate the original builds.

There is one task instance per selected condition here. The table is an observation about this experiment, not a confidence interval for a model family or a universal ordering of work methods.

Keep price basis attached to the result

The historical cost fields need their accounting labels:

For Sonnet, the recorded pieces/whole ratios are about 2.59 for wall time and 1.94 for list-equivalent cost. Those calculations retain the same model and effort setting. They describe the measured treatment; they do not explain every cause of the difference.

List-equivalent cost is a way to value recorded usage under the experiment's price file. It is not the subscription invoice, incremental cash cost or an allocation of the team's monthly plan. A subscription route can consume scarce quota and operator time even when it adds no charge per individual call.

The Codex whole metrics contain a zero cost field, but the historical report says usage was lost when its completed call was killed. Its cost is unknown. I would not put that zero into a cost ranking or describe the run as free.

Interrupted calls also made some recorded totals incomplete. The exact dollar ratios therefore belong beside their source conditions and missing-data notes. They are useful for understanding the work pattern without becoming a promise about what another customer's next build will cost.

A large local suite can measure a small part of the product

Sonnet's pieces arm completed nine of fourteen pieces. It had 1,025 passing local tests while the hidden suite recorded 39 of 60. The report describes missing retry/dead-letter behavior, rate limiting, metrics and signal handling.

The whole arm recorded 198 passing local tests and 60 hidden checks. That contrast does not make tests an obstacle. It shows that the local suite and the acceptance suite were answering different questions, while both consumed the same finite delivery budget.

A well-tested admission path cannot substitute for a missing worker lifecycle. A tested record insert does not establish concurrent idempotency. A happy-path delivery does not establish redirect refusal, backoff, secret rotation or recovery after a lease expires.

I want a piece's acceptance criteria to identify the larger invariant it contributes to. The local result needs a join condition with the next component. Otherwise, a team can accumulate accurate claims about individual parts while the promised customer journey remains incomplete.

That is also why test counts should stay attached to their scope. Passing 1,025 checks and passing sixty acceptance checks are different measurements. Adding them would obscure the boundary that explains the result.

Give the delivery state machine one accountable owner

The webhook task makes the integration problem concrete.

Admission decides whether a request is new or an idempotent replay. Storage establishes durable event and delivery identities. A worker claims a delivery. Retry logic decides when another attempt is eligible. Completion records must belong to the attempt that still owns the work. Shutdown and crash recovery must agree about what remains unfinished.

Dividing those responsibilities across pieces can work, but their state transitions need one coherent contract and an accountable integration owner. If each piece invents its own meaning of “pending,” “leased,” “failed” or “done,” later testing has to discover the disagreements after code has already formed around them.

One independently reported finding in the experiment concerned a stale worker result overwriting a reclaimed delivery. That is a state-ownership problem. A useful contract would name the claim identity that a completion must present and the conditions under which that identity stops being current.

It must also distinguish database ownership from external effects. Preventing a stale completion from updating a row does not undo an HTTP request already sent by the old worker. The attempt, effect identity and reconciliation path need to remain explicit.

For a follow-up decomposition, I would assign shared delivery-state ownership before allocating admission, worker and observability work. The components can then implement against a common transition contract instead of negotiating it indirectly through successive patches.

Decompose around interfaces that can be accepted independently

A useful work boundary has an input, an output, an owner and invariants that another component can rely on.

For the admission component, I would require the tenant and idempotency contract, durable returned identities, replay semantics and the evidence that concurrent duplicate requests do not create duplicate intended work. The delivery-state component owns claim and completion transitions. The worker consumes that maintained interface rather than importing its own interpretation of the database.

Observability can be a separate component when its interface exposes the actual durable states and identities. It should not invent a second accounting system that disagrees with the worker about whether a delivery is still unknown or retryable.

A proposed work packet can make the contract visible:

{
  "component": "delivery-worker",
  "stateOwner": "delivery-state",
  "inputs": ["tenant", "deliveryId", "claimId", "attemptBudget"],
  "postconditions": ["result belongs to current claim", "uncertain effects remain reconcilable"],
  "integrationOwner": "named release owner",
  "evidence": ["valid delivery", "stale completion refused", "interruption recovery"]
}

This is a proposed follow-up packet, not a recovered specification or a newly implemented worker. Its purpose is to show what the assignment must preserve. The next experiment should publish those contracts and the dependency graph before any builder starts.

Separate coherent implementation from independent hardening

The historical mutation results add a real trade-off.

Luna's whole build caught four of twelve reviewer-designed mutations; its pieces build caught seven of ten. Terra's recorded fractions were three of thirteen and eight of ten. Sonnet's fractions were nine of eleven and six of six. Opus pieces caught ten of ten and received the strongest recorded rubric score, but its whole-task counterpart was void.

Those fractions came from different reviewer mutation sets with different denominators. They suggest a test-strength dimension worth examining; they do not establish that the method caused a precise percentage improvement. A difficult twelve-fault set and a different ten-fault set are not interchangeable measurement instruments.

The useful follow-up is to preserve coherent implementation while allocating a separate hardening phase against independent acceptance and fault cases. That phase could be performed by another model or engineer, with the original requirements intact and an explicit budget for correcting findings.

I would measure the complete accepted result after that hardening phase. A whole build followed by independent testing is a different treatment from a whole build stopped at its first green suite. Its review time, rework and elapsed cost need to be included before claiming an operational improvement.

Repeated context and checks are part of the treatment

The recovered token fields make the repeated-work cost visible.

Sonnet whole recorded 11,460,999 input tokens, including 11,182,504 cached input tokens. Sonnet pieces recorded 18,329,783 input and 17,488,997 cached input. Their recorded cache-write fields were 278,367 and 840,360 respectively.

Flash whole recorded 7,713,989 input and 7,216,675 cached input. Flash pieces recorded 31,997,537 input and 28,730,364 cached input, alongside substantially more output. Cached input is part of the recorded input total, not an additional token population to add on top.

The harnesses also differed across model families. The Claude arms used claude -p with a controlled tool/settings setup. Gemini used a custom function-calling loop. The raw model_calls field aligns with builder invocations in this experiment; it should not be casually relabelled as individual provider inference requests.

Those facts explain why I prefer the within-model method comparison here. Even then, repeated setup, checking and context shape are parts of the treatment, not factors that have been held out of the measurement. They are exactly the operating costs a decomposition policy needs to account for.

Reserve integration capacity before the last piece

A fixed deadline makes ordering consequential.

The historical report places much of the missing operational work late in the serial path: rate limiting, metrics, shutdown and documentation. If the budget is exhausted before those assignments finish, the system can be incomplete even when earlier pieces are locally polished.

I would reserve capacity for integrated acceptance at the beginning. The owner needs to know which dependencies can move independently, which transitions must be demonstrated early and what happens when the remaining budget becomes smaller than the remaining integration work.

A useful checkpoint is an end-to-end vertical path with deliberately limited scope. It can expose interface disagreement while there is still time to change the design. That differs from declaring the product complete after a happy path; the remaining boundary and failure cases stay explicit.

The product decision might then be to reduce scope, stop another local enhancement or allocate a separate reviewed hardening attempt. The key is that integration work has an owner and a budget before it becomes the last unresolved obligation.

Keep the missing runs in the evidence record

The planned experiment had fourteen arms. Three never produced usable output: Codex pieces, Opus whole and Gemini Pro whole. The retained tool result also contains empty records for those directories.

The fifteen recovered metric rows are a different population. Four have no hidden score because they are earlier or interrupted repeat runs. Three planned void arms and four unscored rerun rows can both be true. Combining those counts would misdescribe the experiment.

A host-wide kill interrupted several calls, and its cause remained unknown in the report. Scoring environments also differed around DATABASE_URL, and the report recorded about one hidden-check variation between some arm and reviewer runs. Flash whole had no completed blind review record in the recovered report.

These are reasons to preserve the observed cohorts and avoid broad causal claims. A void is not a zero-quality model result. An unreviewed 60/60 build is not equivalent to an independently accepted production release. A repeat is not an independent replication merely because it has another directory.

Turn the result into the next decision

For my next comparison, I would keep the task, template, harness, model and acceptance contract fixed. I would compare coherent implementation plus independent hardening with contract-based decomposition plus a named integration owner. Each treatment would get the same complete delivery budget, including review and rework.

The outcome would include useful accepted behavior, remaining findings, independent fault coverage, elapsed time, recorded usage and qualified owner effort. Missing or interrupted runs would remain in the record with their own state. The actual assignment sequence and dependencies would be retained alongside the executable fixture.

Until that comparison exists, the current data supports a narrower operating choice: avoid assuming that more serial pieces create more useful control. Decompose where the interfaces, state ownership and acceptance conditions are clear enough to compose.

Choose one piece in your current plan. Identify the end-to-end invariant it contributes to, the owner of its shared state and the evidence needed at integration. If those are still unsettled, resolve that contract before adding another handoff.

How this was verified

The historical report was recovered verbatim from its original session record, with the source line and record digest retained. Fifteen slim metric receipts are cross-checked against fifteen expanded numeric/harness records from the original tool output. Previously redacted tokens_missing values remain unavailable. The new analysis reproduces four explicit matched pairs and their arithmetic; it does not rerun builders or hidden tests. The original runnable task, piece list, scorer and arm repositories were not recovered at their recorded local or dev 2 paths. Review mutation fractions and rubric observations are retained historical reports with different sets, not newly verified product defects or model rankings. Prices are the dated experiment's accounting basis, not current API prices. The component packet and proposed follow-up are new design advice, not measured treatment arms.

The new cohort analysis initially passed 87 cases but detected 20 of 22 non-equivalent independent mutations; two additional changes were equivalent under the analysis’s rejection contract (receipt 42 b 1 a 43, set 7710 aec). Three consistent fixture cases expanded the exercise to 90 and made the zero-total and exact-integer guards decisive instead of letting a later guard mask them. The same-set follow-up detected 22 of 22 with both equivalents surviving and control exit zero (receipt dba 4417). Analysis source hashes remained identical. These results qualify the new arithmetic, not the original webhook implementations or builders.