A fleet pause must reach the process tree

I could pause the fleet while preparation kept writing to disk.

The job looked stopped from the operator’s point of view, but its worktree preparation had started Git and dependency processes with lifetimes separate from the task. A fleet reconcile could cancel the task context while a background checkout, install or lock wait continued. That meant the scheduler could stop inference without stopping the expensive work that had already begun.

The incident evidence made the distinction concrete. The qualification record captured the original process, its parent Git, a reset process and a child group. One observed process was in D/balance_dirty_pages: kernel uninterruptible sleep while dirty pages were being written. That is not a process that a signal can instantly remove. The practical lesson was not “send a stronger kill.” It was to give the work a cancellation contract before it touches the filesystem, and to be honest about the point where the kernel can still delay cleanup.

Cancellation has to reach every owner

The supervisor already had the task context. The bug was that preparation paths and waits did not consistently use it. A context only governs work that receives it. If a helper quietly switches to context.Background(), the caller’s pause no longer owns that subprocess. The same is true for a mutex or admission wait that cannot notice cancellation: the parent can return while preparation remains queued to start later.

The fix passes the task lifetime through worktree preparation, snapshot restore, replay and dependency installation, and makes the preparation wait cancellable. External commands run in a private process group so cancellation can reach the command tree instead of only the direct child:

func groupCmd(ctx context.Context, name string, args ...string) *exec.Cmd {
    cmd := exec.CommandContext(ctx, name, args...)
    cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
    cmd.Cancel = func() error {
        return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
    }
    cmd.WaitDelay = 10 * time.Second
    return cmd
}

The group matters because Git can launch helpers, and package installation can create a process tree. Killing only the immediate process may leave descendants holding files, consuming I/O or keeping pipes open. A private process group scopes the cancellation signal to the work owned by that preparation attempt. WaitDelay bounds how long the parent waits for pipes after cancellation; it does not make a process in uninterruptible kernel sleep disappear on demand.

The caller must pass the actual task context too. The qualification deliberately changed the real supervisor call from PrepareWorktreeContext(ctx, ...) to a background context through a Go overlay. The assertion went red with “fleet pause did not cancel actual supervisor preparation.” Restoring the caller context returned it to green. That is a useful causal control because it removes the exact wiring the fix depends on, rather than merely checking that a helper accepts a context.Context parameter.

A cancelled checkout is not a clean checkout

Cancellation can arrive after Git has already changed files. A task should not receive a worktree that looks reusable but has an incomplete index, a held lock or partial snapshot state. Before checkout can write, the preparation path persists and syncs a marker saying that preparation is in progress. If the process stops mid-write, the next attempt checks for a usable index and lock state. If it cannot prove the worktree is coherent, it returns an incomplete-preparation error that the supervisor turns into a typed hold.

That hold is an important safety behavior. The agent should not be allowed to “fix” an interrupted preparation by resetting the branch or deleting state until something appears clean. A human or deterministic recovery path should inspect the worktree and decide whether to restore, rebuild or quarantine it. Cancellation ends owned work; it does not grant permission to erase evidence or discard user changes.

The test surface therefore has two sides: prove cancellation reaches all owned subprocesses and prove a cancelled filesystem operation cannot be mistaken for a prepared workspace. The recorded owning run covers eight focused cases across preparation, real Git work placement and snapshot restoration; it completed in 0.474 seconds. The qualification also records checks for a missing index and a typed hold. The duration is test runtime, not time to terminate the original process tree.

Cancellation is not rollback

I keep three outcomes separate in the operator view: cancellation requested, owned processes exited, and workspace safe to resume. They are related, but one does not prove the next. A context can be cancelled while a child is still responding to termination; a process can exit after partially changing the checkout; and a marker can correctly say “incomplete” even after every child is gone. Collapsing these into one green “paused” badge hides exactly the state the operator needs to choose recovery.

The process-group boundary answers who to signal. The marker, index and lock checks answer whether the files can be reused. The scheduler state answers whether capacity is reserved. Those are different facts with different owners. A safe retry should first resolve the worktree state, then prepare again; it should not infer that cancellation reverted Git or package-manager side effects. If the kernel holds a task in uninterruptible I/O, report termination as delayed instead of claiming the process group stopped cleanly.

This also changes incident timing. Measure from the pause request to child exit and separately from the pause request to a validated reusable workspace. If the second duration is longer, the system is doing useful safety work rather than silently restarting on uncertain files.

The dashboard needs more than Running

The same investigation changed how I interpret scheduler status. In this runner, Running can mean the task has reserved capacity. It does not prove the model has started. Preparation may still be waiting on a lock, cloning files or installing dependencies. If an operator reads that status as active inference, they will misdiagnose a pause, throughput or disk event.

For operations, I want separate states for queued, admitted/preparing, model-active and integrating. Then a fleet pause can say which work was cancelled, which preparation was held, and which task had actually consumed model time. This is a distributed-systems point: the control plane’s state transition is not proof that a side effect has reached the worker or its child processes.

A reliable pause is a contract across owners: the scheduler stops admitting work; the task context reaches every wait; child processes are grouped and terminated; the filesystem records whether preparation completed; and resume rejects uncertain state. If one layer is omitted, “paused” can become a label over continuing disk work.

I would test this on a disposable job by pausing during a lock wait, during Git checkout and during dependency installation. Observe the process group, filesystem writes, marker and later resume decision. Then inject the caller-context omission and require the cancellation assertion to fail. Finally, test the case where a child is stuck in kernel I/O and report that as delayed cleanup, not a successful instantaneous stop.

That is the operating rule I took from this: cancellation is not an API call at the top of the stack. It is ownership that has to propagate down to the side effect and back up as a state the next attempt can trust.

How this was verified

The pinned gorunner qualification at 05defa3bf3237c7f38b44f3594ed70b5066dc462 records eight owning cases passing in 0.474 seconds, a caller-context omission that failed with the intended supervisor assertion, and a restored passing result. It labels the candidate FINAL_CANDIDATE_SOURCE_QUALIFIED_NOT_INSTALLED; this article does not claim the change was installed or that the original nine startup trees were recovered. The receipt records a process in D/balance_dirty_pages, so kernel uninterruptible sleep can defer termination despite correct signal delivery. The writing pass checked the pinned Git objects and receipt; it did not rerun the Go suite or perform a live cancellation exercise.