Distributed systems and durable ownership / Field essay 37

Retire worktrees by consumer lifetime, not terminal task status.

A terminal task state is only a candidate; safe worktree retirement requires the last consumer to release a shared lease, a fresh owner snapshot and verified private recovery outside the target.

Published

DISTRIBUTED SYSTEMS, OWNERSHIP AND SAFE CLEANUP
FIELD ESSAY 37 / DISTRIBUTED SYSTEMS AND DURABLE OWNERSHIP
THE OVERVIEW FILMWatch the full argument

Retire worktrees by consumer lifetime, not terminal task status

The supervisor was dead. Its Git child still owned the checkout.

That is the difference between a task dashboard and a safe retirement protocol. A runner can report a task as finished while a child process still has the worktree open and is writing to it. If cleanup trusts the status row, it can remove the very state the child is using—or discard the only copy of private work.

I designed the retirement boundary around the last real consumer. Native workers take a shared advisory lock on a private lease file outside the checkout. The retirement process must acquire an exclusive nonblocking lock on the same inode. A terminal status makes a worktree a candidate; the lock and the recovery checks decide whether it is eligible.

The supervisor is not the lifetime boundary

A supervisor can exit, crash or be killed while a Git command it launched is still active. The Git process may continue reading objects, updating refs or writing the index. If the parent closes its descriptor, that does not mean the child has stopped holding the underlying open-file description.

The native launcher therefore passes the lease descriptor to direct child processes whose command or arguments refer to that worktree. The parent keeps its own lease through Start and Wait. When it closes, it closes its descriptor; it does not call LOCK_UN, because an explicit unlock would release the shared lock for inherited copies too.

func (l *worktreeLease) Close() error {
    l.once.Do(func() {
        worktreeLeases.Lock()
        defer worktreeLeases.Unlock()
        delete(worktreeLeases.held, l)
        // LOCK_UN would also unlock every inherited copy of the description.
        l.err = l.file.Close()
    })
    return l.err
}

This is intentionally narrower than “process safe.” It covers direct children that preserve the inherited descriptor. A descendant that deliberately closes it needs a separate process-reference audit. I would not represent an advisory file lock as a universal process sandbox.

The ABI makes identity part of the decision. It records the canonical worktree path plus the device and inode of the lease. The file must be regular, owned by the current user, mode 0600 and single-linked; the private directory is 0700. Symlinks, aliases, replaced inodes, malformed metadata and ambiguous state are refusals. The metadata directory lives outside the checkout, so the checkout cannot contain the lock that decides whether it may be retired.

A lock protects data only if recovery is real

An exclusive lock says no cooperating consumer currently holds the lease. It does not prove that deleting the worktree is safe. Retirement also needs a recoverable private copy and a current ownership decision.

The protocol first preserves the public recovery branch and a separate private pack, bound to exact source identity. It archives outside every target tree. Then it acquires the exclusive lease and refreshes authoritative state. The order matters: a snapshot taken before packaging can become stale while the archive is being written. The code rechecks ownership after that work, and any changed head, missing pack, unknown publication dependency, unreadable process reference or interrupted intent becomes a hold for reconciliation.

Resume has its own one-shot rule. A pending private restore can be consumed once when a worktree is recreated. A later resume must not replay the old restore over newer private changes. The restore receipt and the new owner state must agree before the system proceeds.

That gives cleanup a state machine with more than “done” and “not done”:

  • Candidate: task state is terminal, but no destructive effect is allowed yet.
  • Held: lock, owner, archive, source or dependency evidence is missing or ambiguous.
  • Eligible: the exclusive lock is held, a fresh snapshot agrees, and verified recovery is outside the target.
  • Retired: the narrowly admitted operation completed and its result was recorded.

The useful operator rule is simple: uncertainty moves the worktree to held, never to eligible.

Keep qualification and activation separate

This protocol came from a real gap. The initial qualification recorded a checkpoint Git child that survived supervisor SIGKILL without retaining the lease. That was a missing consumer route, not a cosmetic test failure. The later qualification added the actual checkpoint seam, child lifetime tests, current-owner refresh, private-restore checks and omission controls for scheduled routes. It recorded 28 focused Go top-level cases with 19 subtests, 15 focused Python tests, and a set of physical negative controls that removed key protections and had to fail.

The overall suite was not green: the qualification recorded 1,296 passes, one failure and one skip. The remaining failure was attributed to the same targeted baseline timing test, so the result was not presented as a clean full-suite pass. I keep that distinction because a focused protocol result and a repository-wide clean result answer different questions.

The receipt also says the protocol was ready for independent review, not active in production. The normal installer had started, but the paired consumers were not yet qualified or installed in the cited snapshot. Retirement policy was still disabled. No checkout was deleted and no disk savings were claimed by that record.

That boundary is part of the design. A strong test suite can qualify a mechanism; it cannot silently turn on a policy or prove an installation that did not finish. The deployment record needs to confirm both consumers, their hashes and the effective policy before retirement is enabled.

The test I would run before enabling cleanup

Make the child outlive its supervisor in the exact path that performs checkpoint Git work. Kill the supervisor while the child holds the lease, then attempt retirement. The attempt must refuse without changing the checkout. Let the child exit and prove the exclusive lock becomes available. Change the archive after its digest is recorded and require a hold. Change the authoritative owner snapshot after packaging and require a second refresh to catch it. Finally, consume a pending restore once, make a newer private edit, resume again, and prove the old archive does not overwrite that edit.

Those probes exercise the race at the boundary where cleanup becomes destructive. A unit test of the lock helper alone cannot establish that every launch route passes the descriptor or that the retirement command refreshes ownership at the right time.

Terminal status tells me when to investigate retirement. Consumer lifetime, current ownership and recoverability tell me whether I may do it. Until all three line up—and activation is explicitly enabled—the worktree stays.

How this was verified: I checked the pinned qualification receipt and its source manifest against the dossier; the current Go lease file matches the pinned SHA-256, while the Python file in the current checkout has since changed. The pinned receipt reports focused protocol checks and a baseline-attributed full-suite failure. Its installation snapshot records policy disabled and the paired components not installed, so this article makes no claim of active retirement or reclaimed space.

CONNECTED READINGForward and back
Keep reading

The answer deserves
another experiment.

Back to the journal