Nothing that runs should live where cleanup can reach it

The pattern to watch for is quiet. An agent starts building a tool in whatever writable folder its session gave it. The tool works, so it grows. Service units, hooks and registry entries start pointing at paths inside that folder. Nobody moves it, because moving it is always tomorrow's task. Then a cleanup process, a reboot or a retention policy you never wrote does exactly what that folder was made for, and the system that runs your fleet is gone.

In the case behind this note, it took most of a runnable agent harness in about 80 seconds. The same shape came back weeks later in a smaller form. Both times the root cause was where the files lived, not what deleted them.

What it looks like when it happens

The first sign is an odd error from a guard. An agent runs an ordinary shell command and the hook that checks every tool call allows it. Sixty-three seconds later the next shell call fails with a hook error saying the hook's own script does not exist.

The hook had not rejected the command. The hook was gone. So was most of what it guarded: the ticket CLI, the release loop and the gate scripts, all the source of the main tool, and most of the code that launches and routes agent work. In roughly 80 seconds, most of the runnable harness vanished from the folder it ran from.

That folder was a session scratch directory inside an unrelated repository. Nothing in it was committed.

Afterwards, the count was 232 scratch paths referenced by service units, the registry, hooks and scheduled tasks. 90 of them were missing and 22 unit links were broken. The careful backups made before each change lived in the same folders, so they went too.

Why it happens

Durable state settles where the tool hands out a writable directory. An agent works where it is put. If the session gives it a scratch folder, that is where the first runnable file lands, and every later file joins it.

"Move it later" does not come. In this case there was already a written ruling that the harness was product code, committed, with no scratch. The move still never happened. Each day added more units and more hook entries that pointed into the folder, so moving it got harder.

Backups beside the original are copies, not backups. Anything that removes the folder removes both.

A guard that fails closed can lock out its own operator. This is the part I think about most. The per-call hook checked every shell call, every subagent call and every remote run call. When its file disappeared, each of those calls failed closed. That is usually what you want from a guard. Here it meant the coordinating agent could not list processes, could not check the disk and could not stop whatever was deleting. It could still read files and list directories. Every command that could act was refused.

The coordinator pieced the rest together from reads. A separate worker agent independently noticed the source tree had been emptied, broadcast a warning to stop any running cleanup, reported the disk had been 98% full and advised against restarting a responder service. About three minutes after the first refusal, with the operator's approval, the coordinator wrote an interim hook. Twelve seconds after that it stopped the one deletion-capable process still running and disabled its timer.

Three minutes is not long. But for those three minutes, the only thing that could stop the damage was blocked by a guard whose file was part of the damage.

What deleted it: say "unknown" when it is unknown

The cause in this case is suspected, not proven, and I think the honest record is more useful than a tidy one.

The suspect was a scratch reaper whose purpose was to clear idle reviewer workspaces and build output so the disk gate stayed open. Evidence for it:

• It was running in apply mode against the whole scratch root for the entire window, and only ended when it was stopped after 705 seconds.
• The disk was at 96 to 98%, and available space rose from 29G to 86G inside the window, about 57 GB.
• The worker agent reported reviewer worktrees gone, and those were the reaper's domain.

Evidence against it is just as concrete. The restored reaper code was byte-identical to a backup from the previous day. It only walks reviewer workspace directories. It only removes named cache and build directories, and only after a ready marker, three idle hours and a git ignore check. It never touches the code directories that vanished. The same schedule had run ten times that night with no loss.

The reaper was supposed to write a manifest before each removal, which would have settled the question. It wrote that manifest inside the tree being cleaned, and the manifest does not exist on either machine. The best explanation is that either the running code differed from the restored file, or something else did the deleting. That is an inference, and neither can be shown from what is left, because the deleted tree held the logs.

I stopped needing a verdict early. Whatever did the deleting, nothing that runs should have been in a folder that any cleanup process or agent was allowed to touch. The reaper stays disabled either way, because the fix does not depend on it. For a related note on deletion tools and restarts, see Restart is where a deletion tool can lose the only backup.

Recovery from five layers

Recovery used five sources, layered in order:

The transcript was the only durable record of those 12 hours. An audit found 593 harness edits in it, 442 of them shell writes that needed review. A script re-applied the recorded edits onto the base copies. Four semantic checklists, each listing a behavior change and how to verify it, served as the test for whether the restored code was right. The first commit of the restored harness landed the same night.

Not everything came back. The logs from that hour, the reaper manifest and the full replay report were unrecoverable. One scan suite failed 126 of 128 on the restored tree, so an earlier recorded result of 263 out of 263 was not reproduced. The hook in git was still the interim version. The 15,837-byte original survived only on the second machine.

How to prevent it

The changes that followed are the ones I now recommend by default.

• A plain rule. Anything that runs, or is read by something that runs, lives in a git repository outside scratch, and is committed and pushed after every change. No unit, scheduled task, hook or registry entry may point at a scratch path. Backups in the same folder are not backups.
• A real repository for the harness, with an ignore allow-list that tracks code and never logs, databases or runtime state, mirrored to a second machine and a remote.
• An autocommit hook after tool calls and at session stop, committing and pushing in the background.
• A no-scratch guard before tool calls that refuses to write a unit, scheduled task, hook, settings file or instruction file whose content points at a scratch or temp location.
• Cleanup only from an explicit allow-list, never deleting a tracked file, writing its manifest outside the tree it cleans, with a kill switch, tested against a copy first.
• Per-call gates that are not a single point of failure, with a committed copy, a location cleanup cannot reach and a written way for the operator to act if the gate is missing.

One more check matters more than it sounds. The first no-scratch guard was never actually registered. A working copy was registered six days later by a different install script. For six days the rule existed and the guard existed, and the two were not connected. A new guard is just a file until you watch it fire on the installed path. A related published note is Your hook ran. Did the forbidden action stay blocked?

The same mistake, smaller

Weeks later a reboot wiped the agent runtime's session scratchpad, which by default lives under the system temp directory. Agents had been keeping run kits, review packets, scripts, data tables and sync filter lists there. All of it went. Not the harness this time, but the same pattern: durable state had settled wherever the tool handed out a writable directory.

This time the first step was a small experiment. The runtime's documentation lists an environment variable that overrides its temp root. One fresh interactive session with that variable set wrote its marker file under the new durable root. A control session without it wrote under the system temp directory. The experiment also showed that non-interactive sessions get no scratchpad at all.

The fix had three parts:

• The installer sets the override only when it is absent, and never overwrites a value the user chose.
• Each session gets a private durable work directory.
• An advisory fires when a write would create a script, kit or packet under a temp root or the scratchpad. It never blocks. It only names the durable directory.

Pruning is a dry run by default. It never touches write-ahead logs, transcripts or logs, and it never deletes on its own.

Making that advisory non-blocking was deliberate. A warning that fails to fire leaves you no worse off. A blocking gate that disappears takes your shell with it. Whether a missing gate should fail open or closed in general is still an open question for me. I do know that "fails closed, and lives in scratch" is the worst combination.

Checklist

[ ] Decide where the repository lives before an agent writes its first
    runnable file.
[ ] Lint every hook, unit, scheduled task and registry entry for temp
    or scratch paths.
[ ] Commit and push after every change to anything that runs.
[ ] Keep backups somewhere other than the folder they back up.
[ ] Keep a committed copy of every per-call gate, plus a written,
    owner-approved fallback for when it is missing.
[ ] Cleanup tools: explicit allow-list, no tracked files, manifest
    written outside the cleaned tree before each removal, kill switch.
[ ] Test cleanup against a copy before it touches real data.
[ ] Prove each new guard fires on the installed path.
[ ] Record cause as unknown when it is unknown, with evidence on both
    sides, and make the fix independent of the verdict.
[ ] Point agent temp roots at a durable location you control.

A scratch folder is a disk with a cleanup policy you did not write. Treat it that way from the first file.

Go deeper: the Transition2 course covers durable state, hooks and recovery for agent fleets.