The 192-core server was still waiting on one kernel worker

I had a large build machine with far more CPU capacity than its workload appeared to need, yet tasks backed up and alarms fired. Adding cores would not have fixed it. My working diagnosis was a kernel cleanup bottleneck: short-lived sandboxes created cgroup teardown work, and a blocked writeback handover appeared to hold up that cleanup path. I observed the host and traces; I did not independently establish the upstream kernel mechanism, so I keep that boundary explicit.

Capacity is not progress

A host can report low CPU use and still be unavailable. During the incident, commands waited and process cleanup stalled while the machine had substantial headroom in its headline capacity. The useful question was not “How many cores do we have?” but “Which serialized operation is preventing useful work from completing?”

Our workload created isolated sandboxes. That improved the security boundary for commands, but every short-lived environment had lifecycle cost: create a process group, run work, collect output, stop children and release the cgroup. When tests launched real client sandboxes and a timeout killed only the wrapper, descendants could remain. A timeout that fails to kill the full process group does not end the work; it leaves cleanup debt for the host.

My first instinct was to turn off sandboxing. That would have traded a visible slowdown for less containment. The useful response was to stop creating unowned subprocesses and make lifecycle boundaries match the processes they launched.

A queue can turn cleanup into a convoy

The host evidence pointed to work not being scheduled across all available cores. A kernel worker appeared stuck during a writeback handover, while teardown work accumulated behind it. I treat this as the incident diagnosis, not a claim that I independently proved a particular upstream defect. If a shared cleanup worker stalls, more workers may only add more work to the queue.

I reduced churn first. Tests that launched the real client became opt-in instead of running in every ordinary test pass. Heavy tests received explicit lifetimes. On timeout, the whole process group is terminated, not just the parent. For tests whose teardown competes with cached file data, we allow controlled cleanup before removing the environment. These changes addressed the workload we created before touching the kernel.

Then we improved detection. Generic load thresholds can mislead on a large host: a value severe on a small machine may be ordinary on a many-core one. I added signals closer to the failure mechanism—stuck sandbox processes and the specific kernel worker state—so an alert can distinguish busy work from stalled cleanup. Those signals do not replace health checks; they make first diagnosis less speculative.

Replay the alarm against real samples

The first storm alert created urgency, but urgency does not prove the threshold is right. I replayed recorded samples against old and proposed rules. That exposed the trade-off: fewer false critical alarms, with some real wedge cases detected later. We accepted the delay only with a mechanism-specific early warning ahead of the broader critical threshold.

This is how I review an operational alarm. Define the failure it should detect. Collect representative normal and incident windows. Replay both policies on the same observations. Examine the cases that changed. Then ask what additional signal preserves early warning for the failure you care about. A threshold that quiets a dashboard by delaying every alert is not a good retune.

Fix the producer, teardown contract and signal

I changed three layers. Tests stopped creating real sandbox workloads by default. The runner began killing complete process groups on timeout. The host guard gained indicators for the stuck cleanup condition and was retuned against recorded samples. We also upgraded the kernel while retaining a known fallback, but I describe that as an operational mitigation after the observed incident, not proof of a verified upstream bug.

That sequence matters. If we had changed only the kernel, leaked process groups could have recreated the pressure. If we had disabled sandboxes, we would have weakened isolation. If we had simply raised the load threshold, we could have hidden a real wedge. A layered fix reduces workload, makes teardown bounded, detects the failure mode and preserves a rollback path.

Inspect your capacity alarms beside the slowest serialized lifecycle path. If a host has many cores but one cleanup queue, that queue owns progress. What operation in your fleet is serialized even when the dashboard says you have capacity?

A useful host diagnosis has a sequence

I start by separating saturation from blockage. Check CPU use, runnable queue, memory pressure and disk wait, but also inspect uninterruptible processes, process groups and cgroup teardown. If a command timed out, identify whether the parent, children and sandbox all exited. A host can have idle CPU and still have one serialized worker holding up cleanup. The evidence should show both the process that is stuck and the work accumulating behind it.

Next, reduce the source of churn without weakening isolation. Make integration tests that launch real sandboxes opt-in, cap how many can run concurrently and ensure timeout handlers kill the entire process group. Observe whether the cleanup queue recovers under the same representative load. Only then change kernel, mount or host-level settings, with a known rollback and a maintenance window.

Finally, replay alarm rules against the exact incident trace. Count false criticals and measure how much later the real failure would alert. If you move the critical threshold later, add an earlier signal tied to the mechanism. Keep a record of what is measured, what is diagnosed and what remains an inference. What low-level queue would you inspect before adding more hardware to a host that has stopped progressing?

Keep diagnosis, mechanism and upstream cause separate

A useful incident report has layers. First record what operators could observe: work stopped advancing, which processes remained, which worker state persisted, and whether cleanup completed. Next state the mechanism that best fits those observations, such as a serialized cleanup path. Only then discuss an upstream cause if a source-level trace or specific upstream change proves it. I have evidence for the local symptom and mitigation; I do not have a verified upstream commit for the kernel explanation.

That boundary changes how I choose a fix. I can reduce the workload that creates cleanup, ensure timeouts terminate the complete process group, add alerts tied to the stuck state and replay thresholds against recorded samples without claiming to have repaired the kernel. If those changes restore progress, that supports the operational mitigation. It still does not establish which upstream defect caused the event. I keep the host upgrade and fallback as risk management, not as causal proof.

This is a valuable discipline for any infrastructure incident. Separate observations, hypotheses, interventions and outcomes in the incident record. Then future operators can revise the diagnosis without losing the evidence. A causal claim should identify what observation would falsify it and what upstream source would confirm it. Otherwise a plausible low-level explanation can harden into folklore and send the next team toward the wrong subsystem.

Make the alarm explain the next action

A useful alert should tell the on-call operator which lifecycle boundary is stuck, what evidence supports that classification and which low-risk mitigation is available. CPU percentage is context, not an instruction to add hardware. A process count is also incomplete unless it distinguishes expected work from children left behind after a timeout. I keep the early mechanism-specific warning separate from the broad critical threshold so a retune that reduces noise does not erase the first signal of a real wedge. The runbook should name the evidence to collect before cleanup, because restarting too early can erase the state needed to refine the diagnosis.

How this was verified

I reviewed the committed host guard and kernel operations note and kept the root-cause wording at the level those records support. I could not verify an upstream kernel commit, so this article does not claim a confirmed upstream defect.