Tests that spawn processes need a group, a cap and a stop order
The pattern to watch for: a test that exists to prove your code cannot loop. It passes when the guard works. When the guard is missing, in a mutant or in a working copy that does not have the fix yet, the test stops testing a loop and becomes one. Its timeout fires, it reports a failure and moves on. The processes it started do not.
A loop test without its own process cap and its own process group can leave tens of thousands of orphans behind. In the case behind this note, load on a dev machine climbed to 54 and the process table held about 49,000 copies of one command, in three chains, every one parented to init. Nothing that had started them was still alive.
Why a loop test is the most dangerous test you have
During a migration from a legacy command-line tool to a Go binary, the new binary could hand some calls off to the legacy path. Hand-offs like that can recurse. The product's guard is a hop bound: a counter carried through the call chain that refuses past a fixed number of hops. The product bound is 32.
A test existed to prove that guard worked. It built a disposable release in a test cache and exercised the hand-off. With the guard present, the chain stopped. With the guard missing, nothing in the test stopped it.
Two ordinary practices put the guard-missing case in front of that test:
- Mutation testing. Mutants deliberately remove or weaken guards to prove the tests are not vacuous. A mutant that removes the hop bound is exactly the code that will run away.
- Parallel working copies. Three copies of the same workflow ran the test: the main one and two review copies. Any copy without the fix ran the unbounded version.
So the test was most dangerous in exactly the case it existed to catch. I think of any test that spawns processes as running untrusted code on purpose. It needs containment the way an agent does.
Why the timeout did not help
The test ran the binary with a 15-second deadline. When the deadline expired, Go's standard context-bound command killed the direct child and nothing else. By then the child had a child, which had a child. Killing the head of a growing chain does not stop the chain. The descendants were re-parented to init and kept going.
The test then reported a timeout, which looked to the harness like a test behaving correctly.
Stopping the workflow did not help either. That stopped the agents and their test runners. The processes doing the damage were no longer descendants of anything the workflow owned. They had no session or group tying them to the run, so nothing in the run's cancellation path could reach them.
That is a different failure from cancellation that fails to reach work the supervisor still owns. Here the processes had left the tree entirely. A stop that only covers live descendants leaves orphans behind. For supervision and intervention more broadly, see a related note on agent supervision. Orphans are the case where the supervisor has nothing left to reach.
The cleanup order that works
The cleanup took three steps, and the first one failed. The order is the lesson.
- Kill rounds on exact process ids. The processes were matched by full path, to be sure only the test-cache copies were hit, and repeated stop and kill rounds were sent. They could not keep up. Each round killed what it found, and by the time it finished, the survivors had started new processes the round had not seen. A kill loop racing a process chain can lose, and with that many processes it did.
- Remove execute permission from the disposable test copies. This step worked. It killed nothing. It made the next start fail, so the set of running processes could no longer grow. What was running became a fixed set.
- One more kill round. Zero left. Load was 19 and falling a few minutes later.
The incident window was roughly a quarter of an hour. There is no per-step timing record, so I will not put numbers on each step.
Step 2 is the one I keep coming back to. Killing processes did not stop the growth. Taking away the ability to start new ones did, and after that the cleanup was easy. A kill round can finish against a fixed set. It cannot reliably finish against a set that keeps growing.
That step was only safe because the binary sat in a disposable test cache, where changing permissions harmed nothing. I would not do it to a shared system binary. I would rather get the same effect from a limit already in place than from an improvised permission change.
| Lever | What it does | When it works |
|---|---|---|
| Kill by process id | Removes processes it can see | Only against a fixed set |
| Stop the workflow | Cancels owned descendants | Only while processes are still descendants |
| Revoke the ability to start | Freezes the set | Against growth, on disposable copies |
| Process-count limit | Caps the set automatically | Always, if it is in place first |
| Kill the process group | Removes the whole tree as a unit | If every process was started in the group |
How to prevent it
Three conditions were set before the workflow could resume.
A process group per test binary. Start the command as the leader of its own process group, and on timeout and at test cleanup kill the whole group, not one process id. The test runner now does exactly that. It kills the whole group when the leader exits, when the timeout fires, and again at cleanup if neither of those ran.
The runner has its own control test. It starts a script that leaves a sleeping grandchild behind, lets the timeout fire, and fails if the grandchild is still alive five seconds later. Its failure message says the runner killed only the direct child. That test would have failed against the old runner.
A bound on the process count, enforced outside the code under test. The options listed were a per-process limit on the number of processes, a transient control group with a process ceiling, or a hop counter in a file that fails at a fixed number. The test took the third option. The test's own wrapper counts hops in a file and refuses past 20, with a distinct exit code and marker, regardless of what the product does. So a broken product bound fails the test instead of running without limit.
That cap has a control too. It starts a chain where the product's bound of 32 is out of reach, and asserts that the test's own cap of 20 is what stops it. That proves the cap actually fires.
The same guard for mutation runs. Mutants that remove the bound need the strictest limits of all, because of everything you run, they are the most likely to loop.
One review finding changed the test as well. The test had read the product's hop constant to decide what to expect. A mutant that multiplied that constant by 100 survived, because the test simply followed the constant up to a size big enough to take the machine down. The bound is now pinned in the test as the literal 32. Changing it is a review decision, not a test update.
What is not yet confirmed
I want to be clear about the gaps, because a guard that is only planned is not a guard.
- I found no kernel process-count limit around this test or around the mutation runner. The per-test guard that landed is the hop-count file plus the process group. Whether mutation runs have a process ceiling is unverified.
- A machine-level ceiling on processes for the user's session, provisionally 200,000, was prepared later with a drift check and tests. Its own result notes say it was ready for security review and was not a claim of installation. Whether it is installed is unverified.
- Even if it is installed, 200,000 is above the 49,000 seen here. That comparison is my inference, not a measurement. A ceiling at that level keeps the machine usable. It would not have stopped this test early.
Apply agent limits to agent tests
I recommend that the model proposes and code acts, with limits in code and permissions rather than in instructions. For agents that drive a desktop or a shell, I recommend isolation with hard limits, so one runaway step cannot take the environment down. It is easy to apply both to agents and forget the tests the agents run.
The test harness has to contain the code under test the way a sandbox contains an agent:
- Ownership. Every process a test starts belongs to a group the test can kill as a unit.
- A count limit. A ceiling on how many processes can exist, enforced outside the code under test, sized to the test rather than the machine.
- Strictest limits for mutants. Mutation testing produces broken code on purpose.
- A stop button with two levers. One to kill processes, and one to prevent new ones. Better still if a process limit pulls the second lever automatically.
For the broader idea of testing invariants across the whole system rather than one component, see Build the test harness around cross-system invariants.
Checklist
[ ] Start every test that spawns processes as the leader of its own
process group.
[ ] Kill the whole group on timeout, on leader exit and at test cleanup.
[ ] Add a control test that leaves a grandchild behind and fails if it
survives the timeout.
[ ] Run test suites and mutation runs under an enforced process-count
limit sized to the test, not the machine.
[ ] When a test checks for a loop, give the test its own cap,
independent of the product's bound, plus a control proving it fires.
[ ] Pin safety bounds in tests as literals, so raising a product
constant cannot make the test more dangerous without review.
[ ] Write the emergency stop order down: prevent new processes first,
then kill the fixed set, then confirm zero.
[ ] Make sure stopping a workflow also reaches processes that have left
its tree.
[ ] Verify on the machine that any process ceiling is actually
installed, rather than trusting the change that prepared it.
Go deeper: the Transition2 course covers putting hard limits in code and permissions, for agents and for the tests they run.