Building the harness / Field essay H5

Delivery you can recover.

Journal before the risky step, lock the scope, refuse blind retries and leave one checked way out; then never delete agent logs you have not verified saved.

Published

DELIVERY AND RECOVERY
FIELD ESSAY H5 / BUILDING THE HARNESS

Delivery you can recover

If delivery dies halfway through, can I tell what state I am in and get out without guessing? Yes, if the tool wrote down what it was about to do before it did it, took a lock so two runs could not collide, refused to retry blindly, and left me one explicit, checked command to settle the mess. The same rule applies to the agent logs that explain a change afterwards: never delete what you have not verified you saved.

This is the last of five essays about a small open-source harness, pinned to one commit of Agent Harness, 504d930; the guide to the whole flow is HARNESS.md. The fourth essay covered guards at the shell. Delivery is where good reviews go to die: a reviewer approves a feature, someone pushes it by hand, the branch name is wrong, the checkout is dirty, and the code that lands is not the code that was tested. Everything below ran against scratch repositories, a throwaway home folder, and a local bare repository standing in for GitHub. Nothing touched GitHub.

Publish only the commit that was reviewed

A push is one command, so why three helpers? Because delivery is several decisions that have to happen in order. Publish only the commit that was reviewed. Record a prepared generation on disk, so a crash can resume. Then finish: run a check, and only then touch the original checkout. In the real flow, prepare pushes the feature branch and opens a draft pull request, and finish waits for that request to be merged; no command merges a remote pull request for you. That decision stays with a person.

The publisher never forces a push. It refuses the default branch, a branch that is not the one checked out, a head that moved since the final review, and a dirty tree. I ran it against the local bare repository:

== publish_feature.py: a local bare remote stands in for the host
default branch refused     exit 2  BLOCKED_POLICY: Refusing to publish a protected or invalid branch
stale head refused         exit 2  BLOCKED_POLICY: Source HEAD changed since final verification
feature branch pushed      exit 0  PUSHED: Feature branch published
bare remote now has: feature/add-greeting main 

My toy has 12 tests, and I planted seven bugs, each switching off one rule; all seven failed the suite, one by raising an error rather than failing an assertion, and that still counts. Then the real prepare, with inputs it should not accept. The private pull-request body file at a loose mode is refused with a different status from a policy block, because they are different problems. The remote is a local path, not a GitHub repository, and prepare refuses it and writes no state:

== deliver.py prepare: private inputs must be mode 0600
body file mode 0644: exit 3  RECOVERY_REQUIRED: Private input is missing, unsafe, or exceeds its limit
== deliver.py prepare: origin is a local path, not a GitHub repository
exit 2  BLOCKED_POLICY: Origin is unsupported or fetch and push identities differ

How does it know a remote is GitHub? A small function reads the address. It only parses text, so it cannot prove an account has access. I fed it eight addresses:

https://github.com/Acme/Widgets.git        -> acme/widgets
git@github.com:acme/widgets                -> acme/widgets
ssh://git@github.com/acme/widgets.git      -> acme/widgets
https://user@github.com/acme/widgets       -> refused (Unsupported GitHub repository identity)
https://github.com:8443/acme/widgets       -> refused (Unsupported GitHub repository identity)
https://example.org/acme/widgets           -> refused (Unsupported GitHub repository identity)
https://github.com/acme/widgets?x=1        -> refused (Unsupported GitHub repository identity)
/tmp/local/origin.git                      -> refused (Unsupported GitHub repository identity)

Local delivery is a separate, explicit choice made at prepare, not an automatic fallback. If a safety stop turned into a quiet local merge, the stop would be a suggestion. A policy refusal is not a network outage:

== deliver.py prepare --local-only: the explicit, separate choice to deliver without a pull request
exit 0  LOCAL_READY: Explicit local-only generation prepared
== deliver.py finish --mode local: merge and check in a scratch copy, then move the original checkout
exit 0  LOCAL_DELIVERED: Checked local integration complete; remote publication remains outstanding
feature head is now in the original main: yes
parents of the original main tip: 2
== finish again is a receipt, not new work
ALREADY_DELIVERED - Historical delivery remains present; no new remote work was synchronized

Note the two parents: the tool merges in a scratch copy, runs the check there, and fast-forwards the original to the tested merge commit, so the original's tip is a merge commit and not the feature head. A second finish is a receipt and does nothing new. Publication stays outstanding, and a person still has to publish later.

When the merge dies halfway

Integration is where a crash hurts most, because the original checkout is where real work lives. The real integration script does not merge in the checkout. It merges in a separate scratch worktree, runs the check there, re-checks that the original is still clean, on the expected branch, at the expected commit, and only then moves it by fast-forward.

Before any risky step it writes a journal: a small private file with the base commit, the source and where the scratch copy lives, flushed to disk and put in place atomically so nobody reads half a journal. It also takes an exclusive file lock per checkout and branch. I held that lock from a second process and the real script refused to start, without writing a journal. Then I made a branch with a failing test:

== two integrations at once: the scope lock
second integration       exit 3  BLOCKED_LOCAL: Another integration is active for this checkout and branch 
journals written by the refused run: 0
== a failing check in the scratch merge
bad branch               exit 3  BLOCKED_LOCAL: Scratch check failed {'phase': 'check', 'exit_code': 1, 'recovery_required': True}
original main moved: no; active journals: 1
journal phase: scratch
retry, same attempt      exit 3  RECOVERY_REQUIRED: Unresolved scratch journal requires inspection before new work 
retry, another branch    exit 3  RECOVERY_REQUIRED: Unresolved scratch journal requires inspection before new work 

The original did not move, the journal stayed active, and even a good branch was refused until the journal is settled. That is heavy-handed, and it is a judgment call: a comment in the source says the journal exists to prevent an automatic second merge after an uncertain timeout or an interrupted hook, and the tool cannot know whether an earlier attempt touched anything else. The cost is one command:

== resolving: the original is re-checked, then the journal is retired, not deleted
resolve journal          exit 0  RESOLVED: Tested current checkout accepted as recovery baseline; artifacts retained
active journals: 0; archived: 1
good branch              exit 0  INTEGRATED: Verified feature merged into original checkout {'cleanup_pending': False}
original main moved: YES; active journals: 0; archived: 2
good branch again        exit 0  ALREADY_INTEGRATED: Source is already in original checkout 

You pass the journal's full key, named in the failure output, plus the same check. The script refuses unless the original is clean, on the right branch and at the head you name, runs the check on the original itself, and only then retires the journal to an archive. It deletes nothing. RESOLVED accepts a tested baseline; it never claims the feature was delivered.

Did I crash it, or just fail a test? A failed test is not a crash, so I also ended my own script with the timeout command, three seconds into a check that sleeps for six. That sends a termination signal the script does not handle. It is a stand-in for a crash, not a power cut and not a SIGKILL:

== a crash: end our own integrate_local.py with timeout while its check runs
interrupted process exit: 124
original main moved: no; active journals: 1
journal phase: scratch
retry after the crash    exit 3  RECOVERY_REQUIRED: Unresolved scratch journal requires inspection before new work 
resolve after inspection exit 0  RESOLVED: Tested current checkout accepted as recovery baseline; artifacts retained
another branch           exit 0  INTEGRATED: Verified feature merged into original checkout {'cleanup_pending': False}
original main moved: YES; active journals: 0; archived: 4
original working tree clean: yes

The journal was left behind, the original had not moved, a retry was refused, resolve worked, and a good branch integrated afterwards. A lock taken with the operating system's file-locking call is released when its holder exits; that is documented platform behavior and I did not test it by crashing a lock holder. My toy has 9 tests and seven planted bugs, all caught. One path matters and I did not run it: if the failure comes at or after the moment the original moved, the source reports delivered-but-check-failed, and you inspect the original before anything else.

Never delete what you have not saved

The same discipline protects the logs. Picture disk space running short, a teammate pruning old agent logs, and a month later a change that needs explaining where the transcript is gone. The backup had quietly stopped weeks earlier and nobody was told. Transcripts and the write-ahead log are how you explain a change after the session is gone, so the rule is one sentence: never delete logs you have not verified saved. An earlier essay here, Restart is where a deletion tool can lose the only backup, covers a retry that overwrote the only backup.

The backup script is copy-only. Its header says it never deletes, syncs or purges on the remote. A changed file is not overwritten; the old bytes move into a versions folder named by timestamp. I ran it on synthetic logs, with a local directory as the "remote", three times, truncating one file from 24 bytes to 8 and deleting another locally:

== run 1: copy to a local directory standing in for the remote
log-backup.sh exit: 0
credential-named file copied: no
== run 2: a local file is truncated, the remote keeps the old bytes under versions/
remote copy now: 8 bytes
old copy size: 24 bytes
== run 3: a local file is deleted, the remote copy stays (copy only, no delete or sync)
session2 on the remote: still there

A sync would be tidier, and tidy is how a mistake spreads: if the accident is a local deletion or truncation, mirroring it makes the backup lose data too. The cost is storage, which grows without bound, and nothing here prunes it; retention is a decision someone must own. Files with credential-like names are excluded by a pattern list. A secret pasted into a transcript cannot be found by name, and the list's own comment says encryption at rest is the control for that. I did not test any encryption.

A marker file can lie, so the freshness check is strict. The success marker is written only after every primary source copied, and a skipped run writes a different file that is never read as success. A marker dated in the future is invalid, not fresh. The check has four answers, each with its own exit code:

stale (max age 1 second): exit 1
no marker: exit 2
future-dated marker: exit 2

The stale alert says not to delete or prune the logs until a fresh backup shows. The check also runs at session start by design, which I did not run, so a standing watcher adds a periodic look. It writes an alert line only when the verdict changes, or hourly while it stays bad:

old marker: exit 1
BACKUP ALERT code=1 transcripts=stale bulk=fresh
same verdict again: exit 1, new stderr bytes: 0
alert log events: ['recovered', 'alert']
after a new backup: exit 0
alert log events: ['recovered', 'alert', 'recovered']

And the part that proves the point, restore into an empty scratch directory, checked against the remote by checksum and compared with the live copy:

{"restored_into": "<T>/restore", "src": ".claude/projects", "identical": 1, "grown": 0, "differing": [], "no_live": 1, "files": 2}
restore into a live tree: exit 2, nothing created: yes
restore into a non-empty directory: exit 2

One file identical, and one with no live copy: the log I had deleted came back. It also refused a target inside the live tree, and a target that was not empty. A backup you have never restored is only a hope.

What I did not run

  • GitHub, entirely. No draft pull request, no remote finish, no reconciliation of an uncertain push, no commit-policy refusal. The local bare repository proves the publisher's refusals, not the GitHub path.
  • No real crash. The crash is a termination signal ending my own run. I did not use SIGKILL or cut power, and I did not test the lock being released at exit.
  • Not exercised in integration: delivered-but-check-failed, the wrapper's own baseline recovery, hook and submodule refusals, and the default timeouts.
  • No real remote for the backup. Rate limits, the scheduler units (the installer ran as a dry run only), the desktop notification, the session-start hook, the large-file tier and encryption were not run. The restore ran on synthetic logs, not on anything real.
  • Unbounded growth. The versions folder and the local stage grow without bound; I did not model retention.

The three habits I would copy first are cheap: bind each step to the reviewed commit, save state before acting so a restart resumes instead of guessing, and make every fallback a named choice instead of an accident. Then restore something you backed up. If your delivery process were killed right now, which file would tell you where it stopped?

That closes the series. It started with why an agent cannot grade its own work; the through-line is that a claim needs a checkable record, and that every gate should say which way it leans when it cannot tell.

CONNECTED READINGForward and back
Keep reading

The answer deserves
another experiment.

Back to the journal