The agents used 8% of the host. The polling around them used more

Watch for a busy host where the obvious suspect is the workload. In an agent fleet, the obvious suspect is the agents. When a machine runs hot, the easy story is "too many agents in parallel, so cut agents or buy a bigger box." That story feels right because the agents are the thing you see.

Here is what a measurement showed instead. An orchestrator host with 44 cores averaged 48.9% CPU over nine days, with a p95 of 92% and a mean load of 32. Its root disk sat between 87% and 97% full all week. At the moment of sampling it was busy on 16 to 18 cores. The agent engines themselves, 89 Claude, Codex and model processes, took 3.6 cores in total. That is 8% of the host, plus 6.6 GB of memory.

The report put it in one line I keep coming back to: "They are not the problem; the control plane around them is."

What was measured, and how

No clever tooling was needed. One agent built a report generator on top of the system activity history the operating system was already keeping: nine days for the orchestrator host and seven for a second, larger host, at 10-minute resolution. It added a per-minute sampler on both machines that records cgroup usage per service unit, and a 15-second process census that counts which parents are spawning new processes.

That gave two views at different confidence levels, and the report said which was which at the top, before any conclusions:

the per-service attribution rests on about an hour of per-minute samples taken today; the box-level numbers rest on 7 to 9 days.

That caveat matters. The fork rates and disk numbers below are box-level and multi-day. The breakdown of which daemon used how many cores is a one-hour snapshot.

Why the control plane outgrows the work

Every fleet grows supervisors. Something launches lanes, something reconciles their state, something confirms what shipped, something re-reviews, something draws a status board. Each one is small and justified. None of them does the work.

On the orchestrator host, the per-service view read like an inventory of that supervision layer:

The lane needs scanner is a good example of how this happens. It ran on a five-minute schedule and timed out at four minutes, using about 110 CPU-seconds per run, because it rebuilt its view from the lane logs every time.

None of these is big alone. Together they are several times what the engines used.

The fork storm

The striking number was process creation. Through one week the orchestrator host averaged 600 to 1,000 new processes per second. On the next day that jumped to 4,078 per second, and on the day after to 5,751. Kernel CPU went from 7 to 14% up to 25%. A quarter of the machine was going to creating and tearing down processes.

The census pinned it on git children of two supervision services (about 20 and about 10 per second), plus shell, ssh and timeout children of the re-review runner, and file-lock calls from a sync loop and the lane supervisor. The report's verdict: "That is polling by spawning processes, not computing."

The pattern is easy to fall into. Each service wants to know the state of every lane, and the quickest way to ask is to shell out: git once per lane, per cycle, with an ssh or a lock around it. With one lane that is nothing. With dozens of lanes and half a dozen services polling on short cycles, it adds up to thousands of processes a second. The cross-host version is just as real: 7,412 ssh logins into one host in 30 minutes caused cgroup churn and writeback storms there. Git state shared across a fleet has its own coordination traps, which I wrote about in shared git state is the fleet coordination bug.

One limit of the evidence: the census shows which services were forking. It does not show why the rate jumped fourfold in one day, and no note explains the step. More lanes and pollers coming online that week is a plausible guess, and I label it as one.

The space that never came back

While measuring, two multi-gigabyte "ghosts" turned up. The lane board's in-memory store was holding a 13 GB append-only file open after the file had been deleted, so the space never came back. It was also retrying a failed rewrite ten times a second, which filled the logs and used CPU. Its config and data had lived in a session scratch folder that was wiped earlier that day, so the store was still running on a file with no path. Moving it to a durable location brought back 22 GB of disk.

A process holding a deleted file open is a classic operating system trap, and the usual disk-usage summary will not explain it. It only shows up when you look at disk usage per consumer rather than at a single percentage.

The idle host was not idle

The second host is a much bigger virtual machine with 192 vCPU. By CPU it looked mostly idle: 85% idle on average, a p95 of 52% made almost entirely of short bursts, and 80 to 95% of memory unused. A CPU-only view would call it wasted and start moving work onto it. (A different limit on a host like this is described in the 192-core server was still waiting on one kernel worker.)

The tight resource there was disk I/O. On its system disk, daily p95 utilization ran 72 to 86% over five days, iowait reached 9.8% on one day, and a five-second I/O sample showed 13,000 write IOPS and 9,000 flushes per second. That disk sat on physical storage shared with other critical services.

The shared test database was already set up for throwaway use, which was correct. The flushes came from elsewhere: JavaScript dependency trees in one session scope (31 processes, 8.6 GB), per-lane database containers that were not on the tmpfs pool, and an hourly prune task that used 52 CPU-minutes and peaked at 11.7 GB per run walking thousands of worktree directories.

So the most serious finding was not about load at all. Development tooling on the "idle" box was competing for physical storage that other critical services also depended on, with no I/O limit between them at the time. Memory had the same shape: the guest held a large reservation it was barely using, while the physical host underneath had little left for anything else.

How to prevent it

The report ended with a ranked plan, and almost none of it needed new hardware:

1. Stop polling with fork and exec. Make services event-driven, or batch git calls: one ref listing per cycle, not one call per lane. Target: fork rate back under 1,000 per second and kernel CPU down from 25% to about 10%.
2. Move the control plane to the idle host. The first cut-over was rolled back because a tool and its credential were missing on the target. A move needs a dry run of every dependency.
3. Fix placement defaults that still started lanes on the busy host.
4. Throttle guest I/O so lanes can never starve neighbors on shared storage: a disk throttle on the VM, I/O limits on lane slices, per-lane databases moved to memory-backed storage, and the prune run less often or with time limits.
5. Turn on memory ballooning for the guest instead of shrinking it, so unused memory goes back to the host.
6. Put CPU and memory caps on every daemon, with the sampler as the yardstick.
7. Shrink the busy host only after the control plane moves. The report says so explicitly: not while its CPU p95 is 92%.

The plan only became reviewable once each item had an expected effect next to it: fork rate under 1,000 per second, kernel CPU around 10%, disk p95 under 50%.

The in-memory store was fixed the same day. The rest became tracked work: a watchdog that checks a resource ceiling per daemon (RSS, forks per minute, ssh per minute), a persistent channel in place of per-call ssh, and a measurement "before and after on both boxes."

Here is where the record stops. I did not find a measured "after" for the fork rate, kernel CPU or disk utilization. The plan has targets, the first move was rolled back, and both hosts have since been retired or scheduled for retirement as work moves to new hardware. I can say what the problem was. I cannot give a number for how much the fix helped, and I will not invent one.

How to detect it on your own fleet

Before I point at the workers on a hot host, I split the question into named workloads: engines, daemons, launchers, prune tasks, dependency trees, per-lane databases. "Is the host overloaded?" has no answer until you do. The same host looked overloaded or idle depending on whether you measured CPU or flushes per second.

Then I look at three signals that a CPU percentage hides:

• Processes created per second, per service. Kernel CPU at 25% is a symptom. Forks per second per service tells you who to fix.
• I/O on shared physical hardware. An idle CPU on a guest that shares storage with other critical services is not spare capacity.
• Disk held by deleted files. If usage does not add up, look for open handles on deleted files.

Checklist

• [ ] Engines are measured separately from the supervision layer before anyone proposes fewer agents or more hardware.
• [ ] Every report states which numbers rest on hours and which rest on days, at the top.
• [ ] Per-unit cgroup samples and a process census run on every host.
• [ ] Forks per second per service is a first-class metric with an alert.
• [ ] No service polls by shelling out once per item. One batched query per cycle, or events.
• [ ] Hosts talk over a persistent channel, not per-call ssh.
• [ ] Guests that share storage with other critical services have I/O limits.
• [ ] Every daemon has a ceiling for CPU, RSS, forks per minute and ssh per minute.
• [ ] Each plan item carries an expected effect, and the "after" number lands in the same file as the "before."

Go deeper: measuring where agent infrastructure really uses its cycles is part of the Transition2 course.