The spec, the reviewers and the evidence ledger.

The spec, the reviewers and the evidence ledger

What has to be written down, and who has to read it, before I let an agent call a ticket done? A spec that says what done means before the work starts, two reviews bound to that exact text, and a merge that checks the results on the exact commit it is merging. This essay covers the first two, then the last gate in the chain.

It is the second in a series about a small open-source harness for agent work. I pin everything to one commit of Agent Harness, 504d930, and the guide that describes the whole flow is HARNESS.md. The first essay argued that an author's "it's done" is a claim, not evidence, and showed how a review gate refuses a stale approval. Here I look at what the reviewers are handed, and what the merge has to know.

The failure I am designing against

A ticket closes as done. The tests are green. Someone asks for the one thing the ticket was for, and it does not work. The tests checked whatever the author wrote down afterwards. Nobody ever wrote down, before the work, what done meant.

"Why not just write tests?" Because a test checks what you thought of. A spec is short: an outcome, a scope, numbered acceptance criteria, negative cases and risks. Every criterion ends with a Verify by: clause, which names the check before the code exists. Every negative case names the criterion it threatens, so you list the inputs that must be refused. It is an order slip written before the kitchen starts, not a receipt written afterwards. The site has an earlier essay on the same failure from the reporting side, Landed is not done: define done before agents report it.

Lint the boring mistakes first

The dull errors are cheap to catch before the build and expensive after it. I ran the real spec tool's lint on a deliberately flawed spec: a template placeholder left behind, a criterion with no Verify by, and a negative case pointing at a criterion that does not exist. Then on the fixed spec:

== 1. lint a spec with a placeholder, a missing Verify by and a NEG naming an unknown AC
exit 2
FAIL
  - section `## Outcome` still holds a template placeholder like <...>
  - AC-1 has no `Verify by: <check>`
  - NEG-1 must name existing criteria in parentheses (unknown: AC-9)
== 2. lint the fixed spec
exit 0
PASS sha256 of the core: <hash16>...

The real lint is stricter than the toy I wrote beside it: it also wants a source on each criterion and a resource-budget section. The <hash16> is the lab masking a hash that differs from run to run.

Two reviewers, two different questions

One careful person should be enough? They ask different things. The adversarial review asks how this goes wrong or gets abused. The acceptance review asks whether the spec describes what the user wanted and whether each criterion can really be checked. The tool accepts a model's plan report, a native agent that the log confirms really ran, or a person. It refuses the same reviewer in both roles and refuses the author as a reviewer, because a reviewer who shares the author's habits tends to share their blind spots.

The binding is a hash. The tool hashes the spec text above the generated ## Reviews table, and each review records that hash. Change one word and both reviews go stale. Recording a review does not change the hash, because the table is excluded. I recorded two human reviews, and the spec grew its table:

## Reviews

Spec core sha256: `<sha256>`

| Role | Verdict | Reviewer | Kind | Source | Bound to |
|---|---|---|---|---|---|
| adversarial | ACCEPT | Red Reviewer | human | pairing session | current |
| acceptance | ACCEPT | Blue Reviewer | human | pairing session | current |

Then I called the real review_problems() directly, on a clean copy, an edited copy, the same person in both roles and the author reviewing their own spec:

good.md -> no problems
edited.md -> ['no adversarial review of the current spec (earlier reviews are stale: the spec changed after review)', 'no acceptance review of the current spec (earlier reviews are stale: the spec changed after review)']
same person in both roles -> ['both reviews come from the same reviewer/model; get the other role from a different model, a native red-team agent, or a human']
author reviewing own spec -> ['the adversarial reviewer is the spec author']

A model's report has to carry the stage and the hash of the exact packet it read, and the packet is the spec core. I hand-wrote a fixture report naming the wrong packet and it was refused, with a message to review exactly the packet file; the same fixture with the right hash was recorded. That fixture is not a model review. I did not run one.

Planted bugs in the toy

My toy spec checker has 11 tests. Then I planted seven bugs, each removing one rule, and the suite failed every time:

mutant CORE: FAILED (failures=1)
mutant STALE: FAILED (failures=1)
mutant VERIFY: FAILED (failures=1)
mutant NEGREF: FAILED (failures=1)
mutant LATEST: FAILED (failures=1)
mutant SAME: FAILED (failures=1)
mutant AUTHOR: FAILED (failures=1)

The one I care about most is LATEST, the rule that a later block supersedes an earlier accept. It passes every happy-path test, which is exactly why it needs a planted bug to prove it is checked.

The ledger after the reviews

In the real flow the spec and its reviews are attached to the ticket. Each criterion then gets evidence, tied to a test record. An independent person clicks through the deployed build. The main agent signs off, and it cannot be the owner or either reviewer. A done command lists whatever is missing. All of that talks to a ticket system, so I did not run it: new, attach, evidence, click-test, signoff and done are described by the guide and not exercised here.

Can an author invent a reviewer name and approve their own spec? The tool's own documentation says its threat model is careless or hurried agents, not adversarial ones. Names nobody verified are printed under a heading called self-attested, so a person can see them. It makes the honest path easy and the lazy path visible. It does not make cheating impossible.

The last gate: what must be true before a merge

The story behind the merge gate is short. A pull request had two red checks. An owner merged it anyway, because nothing between the owner and the button looked at the results. The fix is a hook that reads the checks on the exact commit before any merge command runs. An earlier essay here, A merge gate must prove the tests ran, covers the neighbouring question of whether the tests executed at all.

GitHub already has branch protection, and the gate does not replace it. Protection lives on the server and may be unset or list only some checks. This gate runs on the agent's side, before the command leaves, and it sees the exact command, which the server never does. So it can insist the command names the commit. It takes the union of branch protection, rulesets and the pull request's own required list. Every check and status on the head commit must be success; skipped or neutral is fine only for a check that is not required; an unknown conclusion blocks, because the allow-list is explicit.

I did not point it at GitHub. I drove it in-process with a fake gh that I wrote, which answers only the calls the gate makes, for one made-up pull request. Here is a part of what came back:

required check skipped         exit 2 BLOCKED
      merge-gate: BLOCKED merge of acme/widgets#7 (head 111111111111).
      - required check 'e2e' concluded SKIPPED (policy: a required check must be success)
optional check skipped         exit 0 allowed
      merge-gate: all checks green on <head>
required never reported        exit 2 BLOCKED
      merge-gate: BLOCKED merge of acme/widgets#7 (head 111111111111).
      - required check 'e2e' has not reported on this head
check still running            exit 2 BLOCKED
      merge-gate: BLOCKED merge of acme/widgets#7 (head 111111111111).
      - check 'lint' is not finished (IN_PROGRESS)
required set unreadable        exit 2 BLOCKED
      merge-gate: BLOCKED merge of acme/widgets#7 (head 111111111111).
      - the required checks could not be read (branch protection, rulesets and gh pr checks all failed)
head moved while reading       exit 2 BLOCKED
      merge-gate: BLOCKED merge.
      - the pull request head moved while the checks were read

Anything the gate cannot determine blocks. That is the cost of failing closed: a wait and a retry. The merge command also has to carry the verified head, so GitHub itself refuses a head that moved after the check:

merge, no head pinned          exit 2 BLOCKED
      merge-gate: BLOCKED merge of acme/widgets#7 (head 111111111111).
      - the merge must carry the verified head: add --match-head-commit <head>
merge, head pinned             exit 0 allowed
same merge inside bash -c      exit 2 BLOCKED
      merge-gate: BLOCKED merge of acme/widgets#7 (head 111111111111).
      - the merge must carry the verified head: add --match-head-commit <head>
merge target not one PR        exit 2 BLOCKED
      merge-gate: BLOCKED merge.
      - merge target is built at runtime; use a literal number or URL
not a merge (pr view)          exit 0, gh calls made: 0

When the checks are flaky and the merge really is urgent, there is a waiver, and it is narrow: it names the repository, the pull request and the exact head, it is single use, and it expires after 24 hours. My first attempt without an approver was refused with exit 3. With an approver it was recorded, the next merge passed despite the red check, and the one after that was blocked again:

waive, no approver             exit 3
      merge-gate: a waiver needs main-agent approval: run it through the main agent so the hook stamps it, or pass --approve
first merge after waiver       exit 0 allowed
second merge, same head        exit 2 BLOCKED
      merge-gate: BLOCKED merge of acme/widgets#7 (head 111111111111).
      - check 'lint' is FAILURE

The report that follows is the audit trail. I am showing its heading and count; the full row also carries the waiver id, the reason and a ticket reference, which I left out.

   Merge gate waivers and decisions
   waivers: 1

The harness documentation describes the waiver log as hash chained: tamper-evident, not authenticated. That would make an edit show, but it would not prove who wrote an entry. I did not tamper with the log, so I have not watched it detect anything; that sentence is the documentation's claim, not something I observed. The report marks the approver as asserted. If you copy anything from this, copy the habits, not the trust model.

What I did not run, and what the gate cannot see

• No GitHub and no real gh. The merge gate ran against a fake client I wrote. I did not exercise the real subprocess timeouts, the REST and GraphQL merge forms, the workflow-file reading, or the waiver stamp path.
• Known limits, from the harness documentation. Merge queues, GitHub tools called over MCP and browser merges never reach a shell hook. A script can hide a merge. An agent that switches CI off can merge with zero checks, which the documentation calls an accepted risk. A subagent can evade the waiver guard by writing a script. I ran two disguises, bash -c and a runtime-built target, and no more.
• No tamper check. I did not edit or corrupt the waiver log, so the hash-chain behaviour is documented, not observed.
• No model review of any spec. The spec reviews above were human-recorded or a hand-written fixture.
• The ticket flow (new through done) needs a ticket system and is only described.

Which of your last three tickets could say, in writing, what done meant before you started?

Next in the series: Escalation rules and the bill.