Agent fleet field notes / Field essay 67

Check that each git worktree resolves to itself.

One shared git setting made 17 linked worktrees write into a live tree. How to detect it with one command, prove the fix with a negative control and keep worktree isolation real.

Published

Check that each git worktree resolves to itself

The pattern to watch for: edits to a live tree keep disappearing. A person or an agent changes a file, and a minute later it is back to what the main branch says. There is no error, no conflict and no log line saying "reverted". Meanwhile a separate checkout that is supposed to track the main branch has stopped moving.

When you see those two symptoms together, check where your linked git worktrees think they live. In the case behind this note, all seventeen of them pointed at the live tree, because of one line in one shared config file.

What the setup looked like

An agent fleet ran from a live tree on a dev machine. That tree had no git directory inside it. Its git directory sat beside it as a separate repository, and git found the live tree through the core.worktree setting.

Several other checkouts hung off the same git directory as linked worktrees, created with git worktree add: a deploy checkout, an ops checkout, a runtime checkout, a set of release trees and the per-task worktrees agents work in. Seventeen in total. The point of linked worktrees is isolation. Each task gets its own files, and nothing one task does should land in another.

Three timer-driven services had a pre-start step that force-checked out the main branch in the deploy checkout before each run. The intent was to keep that checkout pinned to the main branch. They ran every minute.

Why it happens

The repository had extensions.worktreeConfig turned on. With that setting, each worktree can have its own per-worktree config file. The main worktree did have its core.worktree entry in its per-worktree file. But the same core.worktree line was also in the shared config file.

Linked worktrees read the shared config. So every one of them inherited the live tree as its work tree. Asking the deploy checkout for its top level printed the live tree. All 17 linked worktrees did the same.

Now put that next to the timers. "Force-checkout the main branch in the deploy checkout" really meant "force-checkout the main branch into the live tree". Every minute, the forced checkout threw away any uncommitted live edit. The deploy checkout's own files stayed frozen, because nothing was writing to them.

The git-worktree manual covers this case. When per-worktree config is enabled, core.worktree has to move from the shared config into the per-worktree file. Here it had been copied there and never removed from the original place.

Two things make this hard to spot:

  • Isolation is declared, not checked. Seventeen separate directories look like seventeen separate workspaces. Nothing in a directory listing tells you that git resolves them all to one place.
  • Config scope is invisible in the file name. A setting in the shared config applies to every worktree, whatever the file looks like it belongs to.

How to detect it

The detection is one command per worktree:

git -C <worktree> rev-parse --show-toplevel

Compare the output with the path you created. If they differ, that worktree is writing somewhere else.

With per-worktree config turned on, audit where core.worktree is set:

git config --show-origin --show-scope --get-all core.worktree

Any hit at local scope in the shared config file is a bug.

I would run the first check at creation and again in the fleet's regular health probe, because a later script can re-create the bad line at any time.

Prove the fix in a throwaway repository first

The obvious fix was "delete core.worktree from the shared config". That is not a change to try first on the live repository, for two reasons. The live tree has no git directory inside it, so core.worktree is the only thing telling git where it lives. And the autocommit and a path linter both read that setting. Get it wrong and the live tree loses its repository, or the autocommit starts committing from the wrong directory.

So the same layout was built in a throwaway repository on git 2.43.0: a separate git directory, core.worktree in both the shared config and the per-worktree file, per-worktree config enabled, and one linked worktree.

Step Result
Before any change Linked worktree's top level printed the main directory. Bug reproduced.
Remove core.worktree from the shared config only Main directory still resolved to itself. Linked worktree printed its own directory. Fixed.
Negative control: also remove the per-worktree entry Main repository's top level fell back to the current directory. Broken.

The negative control is what made the experiment worth running. Without it, you know removing one entry fixes the linked worktrees. You do not know which of the two entries holds up the main tree. The control showed that the per-worktree entry is what keeps commands with only an explicit git directory working. So there was one line to remove and one line that must never be removed. That is a sharper result than "this fixes it".

The change and the proof

The fix was config only. No files moved. First, the per-worktree setting was written again with git config --worktree, a no-op that states the intended end state. Then core.worktree was unset in the shared config file alone. The only difference in the shared config was the removed line under the core section. The per-worktree file stayed byte-identical. Both files were backed up first, and the backups were checked by checksum.

Within seconds of the change, each dependent was checked:

  1. The main tree still resolved. Asking the git directory for its top level printed the live tree. The autocommit's own invocation, with an explicit git directory and work tree, showed the main branch, ran a status and a staged diff cleanly and still read a repository flag it depends on. Plain status without an explicit work tree worked through the per-worktree file. The path linter still got the live path, now from the per-worktree file.
  2. The linked worktrees were their own again. The deploy checkout printed its own path. All 17 printed their own directory. Three still printed a path under the live tree, because their parent folder was a symlink into it. Those were ignored and untracked there, so the autocommit would not pick them up.
  3. The autocommit kept working. It made two commits on the main branch in the next few seconds.
  4. Status in the deploy checkout reported its own tree, with a detached head, as expected.

What the fix exposed

Fixing the redirect showed how much damage it had done. Every earlier checkout in the deploy tree had written its files into the live tree, while the deploy tree's own index and head kept advancing. Its working files no longer matched its head: the short diff stat reported 4,992 files changed. The ops checkout had the same kind of drift, with 5,167 entries differing from its head.

Those trees were not re-synced as part of this fix. Services were running code from both, so a re-sync would really be a deploy of different code to running services. That needs an owner decision, not a cleanup step. For the same reason, the forced checkouts were not put back into any timer. They come back only with a reviewed test that proves a checkout lands where it claims.

The forced checkouts had been removed from the timers about an hour and a quarter before the config fix. That stopped the reverts but left the cause in place. The redirect still applied to anything that touched a linked worktree. Stopping the symptom first is fine. Calling it fixed at that point is not.

Rollback and the rule

The rollback is one copy of the backed-up shared config, and the written plan says plainly what that would bring back: every linked worktree writing to the live tree again. If a caller breaks after the fix, the preferred response is to give that caller an explicit work tree, not to restore the redirect.

The rule for a repository like this:

  • core.worktree lives only in the per-worktree file.
  • Anything that wants the live tree passes an explicit git directory and work tree, as the autocommit already did.
  • Anything that writes the setting uses git config --worktree. A script that writes it with plain git config core.worktree will re-create this problem.

I wrote about other ways linked worktrees still share state (exclude files, refs, mirror pushes) in Shared git state is the fleet coordination bug. Config scope belongs on that list. For worktree lifetime, see the related note Retire worktrees by consumer lifetime, not terminal task status.

Why I care about this one

I recommend that coding agents work in isolated parallel tasks, because isolation prevents collisions. This is that principle failing quietly. The isolation existed on paper. One inherited setting turned every worktree back into the shared live tree. Isolation you have declared but never checked is not isolation.

I also recommend a negative control for every check, because a passing run proves nothing if it could not have failed. In the throwaway repository, the control that broke the main tree is what showed which line had to stay.

Checklist

[ ] After creating a linked worktree, assert that
    `git rev-parse --show-toplevel` equals the path you created.
[ ] Run the same assertion for every worktree in the fleet health probe.
[ ] With per-worktree config on, run
    `git config --show-origin --show-scope --get-all core.worktree`
    and treat any shared-config hit as a bug.
[ ] Write worktree settings only with `git config --worktree`.
[ ] Never run a forced checkout or any destructive git command from a
    timer against a path you have not verified resolves to itself.
[ ] Prove config fixes in a throwaway repository with the same layout,
    including a negative control that shows which line holds things up.
[ ] Back up config files with a checksum before changing them.
[ ] Write the rollback down along with what it would bring back.
[ ] When a fix exposes drift in running deploy trees, stop and get an
    owner decision before re-syncing. A re-sync is a deploy.

Go deeper: the Transition2 course covers isolated parallel agent work and the negative controls that keep it honest.

CONNECTED READINGForward and back
Keep reading

The answer deserves
another experiment.

Back to the journal