Fourteen control files became one release command

The pattern to watch for

Every gate is green except the one that actually puts bytes on the internet.

That was the state of a release of this site on the morning of 28 September. The release was a frozen archive of 977 entries. Every entry hash had been checked against a manifest. An independent review was done. A preflight against a freshly fetched main had passed. The agent that owned the release had checked all 14 control files against the pinned release packet. And nothing could ship, because the hosting dashboard showed only a loading spinner and the deploy command line on that machine had no authenticated account.

When a release process has more checks than steps that ship, watch for the ceremony itself starting to produce failures. The checks are rarely the problem. The extra state around them is.

This is how the site got from that morning to a release that is one command, and which checks survived the move.

The old regime

The rules for site releases opened with an instruction to check and update fourteen files before any commit: a work log, decisions, architecture, memory, blockers, backlog, in-progress, completed, configs, operations, connections, tooling, a really-tested list and a pending-real-test list. Eight of them also had to be updated before an agent stopped.

Then came the deploy rules:

• Deploy only if the installed preflight exits 0 after fetching main, deploy exactly the verified commit it returns, and record the receipt.
• Verify the exact release archive's checksum and every entry hash against a manifest committed on that commit. Never use the mutable build folder in its place.
• Verify editorial approval, frozen media and source hashes, a production-only build, the hosting deployment result and an independent public readback. "A local build or ZIP is not a live release."
• Publish only exactly approved media and copy.

Every one of those rules came from something that went wrong earlier. Read one at a time, each is reasonable. Together they made a ritual, and the ritual started producing its own failures.

Why it happens

Three failures from that period show the shape.

The preflight assumed a repository that did not exist. The day before, the agent ran the preflight against the site folder. It returned exit 2, "not-a-repo", with no verified commit. The rule required a fresh fetch, a head commit exactly equal to main and a clean tracked tree. A local repository created on the spot would not satisfy it. The rule had been written for a remote-backed repository the site did not yet have. The agent stopped instead of faking one, which was right.

The artifact was huge because the build was not reproducible. The staged archive was 923,359,198 bytes, with 977 files. The agent streamed every entry and compared it with the current build output. The archive passed its integrity check, but six HTML pages differed: the feed, the journal index and four notes from one week. The agent called this local build drift, not proof that the live site had drifted. The cause sat in the next line of the log. The build was date-sensitive and read frozen binary sources from outside the site folder, so putting the site folder under version control would not by itself make releases reproducible. The checks caught the drift. But they were only needed because the system kept producing different output from the same inputs.

The process generated its own noise. One checkpoint records that the first release-verifier test failed because it pointed at a historical default artifact location. It passed once pointed at the current pinned archive, and the agent claimed no pass for the failed run. That honesty was correct. It still meant a cycle went to a failure that came from the process, not from the site.

Each of these is a check doing what it was told. The pattern is that the checks had accumulated faster than the system underneath them got simpler.

The blocker that cleared on an observation

Then the dashboard. The blocker log recorded that the hosting dashboard showed only its loader, and that a read-only authentication probe of the deploy command line was running. Minutes later it recorded the dashboard as still loader-only after project, home and reload probes, and the command line as having no authenticated account. No deployment was allowed until a supported route worked.

The decision log set the policy: keep the working production site in place, publish only the exact release once supported access returned, and use no authentication bypass and no "speculative loader remediation".

A single cache-bypassing browser reload then brought back the authenticated dashboard. The account chooser and the existing account rendered, and no cookies, credentials or session data had changed. The entry I care about most came next: "Previous loader-only blocker is superseded by observed recovery, not by an inferred cause."

Shortly after, the release was live. The archive extraction finished, one deployment ran, and the cold readback passed 71 of 71. The command line was still unauthenticated, and it turned out not to be needed.

I want to be precise about why I like how that was handled. Earlier probes had included ordinary reloads, so "just reload it" was not obvious beforehand. The agent did not invent an explanation for why the dashboard was stuck. It did not touch cookies and did not look for a way around the login. It recorded what it saw: loader-only, then recovered after a cache-bypassing reload. It closed the blocker on that observation and left the cause marked unknown. I still do not know why the dashboard hung. The log says so, and that is the right state to leave it in.

How to detect ceremony overload

These are the signals I now look for in any release process:

• Rules that must be updated before every commit, regardless of what the commit touches.
• A byte manifest that exists because the build cannot be trusted to produce the same output twice.
• A preflight that assumes a layout the project does not have.
• Failures that come from the checking machinery itself, such as a verifier pointed at an old default.
• Every gate green and nothing live. If the only failing step is the one that ships, the gates are measuring the wrong thing.

What replaced it

On 19 October I asked for the release to be rebuilt "to not be so annoying." Videos would embed from an unlisted video host, images would sit in object storage at public URLs, and the content would live in version control, easy to push to the site. Articles should go live "as soon as they are ready".

The new release instructions fit on one screen. The core:

• One command. A single release script handles production. It refuses unless the checkout is clean and exactly main, and prints "refused: commit and push to main first" when it is not. A preview flag deploys the current tree to a preview branch. Either way, the script builds, deploys, compares the served bytes with the build and appends to a release log. "Don't deploy any other way."
• Small deploys. Images and old campaign media moved to object storage, and old media links redirect there. Films are unlisted embeds. Nothing over 25 MiB goes into a deploy. The script's header notes that the upload is now a few MB. The old artifact was 923 MB.
• A frozen baseline. A snapshot of the live site as of 19 October, with its evidence kept alongside. It changes only on purpose and in its own commit.
• Approval is a merge. Each essay is a folder with a metadata file and an article. Setting the publish flag and merging to main counts as approval.
• Retired, not deleted. The 14 control files, the per-release archives and manifests, and the old issue gates in the build are kept for history in a dated file.

The readback keeps it honest. After deploying, the script retries a byte comparison against the public URL for up to about a minute, because the custom domain takes a few seconds to switch over. If the comparison still fails, the script exits non-zero with "READBACK FAILED". Production releases are logged and committed. As of this writing the log holds 111 entries, from 19 October to 5 November. Each records a timestamp, commit, deployment, essays and readback. The latest readback checked 72 paths with no failures.

What survived, and the gap that remains

Most of the checks survived in smaller form. "Clean and exactly main" is the old fetched-main preflight. The served-bytes readback is the old independent public readback. The log is the old receipt. What went away was the extra state, and the reason to pin a 923 MB archive, because the deploy is now small and built from version control. A 55-line shell script that refuses, builds, deploys, reads back and logs does the work the control-file regime was trying to do.

That is the argument in the release gate is the product: a gate is worth exactly what it enforces, and a script enforces it every time where a checklist depends on someone remembering. It also follows the split in separating probabilistic development from deterministic shipping. Agents write. A deterministic script ships.

One gap, stated plainly. A release contract should include a negative control that fails on a planted defect. The old regime ran seven bad-evidence controls. I have not confirmed that the new script has an equivalent. Until I have, the readback is a check I believe catches a bad deploy, not one I have watched catch one.

Checklist

• [ ] One command ships production, and it refuses unsafe states (dirty tree, not exactly main) with a clear message.
• [ ] The deploy is small and built from version control. If you need a byte manifest to trust the build, fix the build first.
• [ ] The build is reproducible: no date-sensitive output, no inputs from outside the tracked tree.
• [ ] A public readback compares served bytes with the build, retries for a bounded window, and exits non-zero on failure.
• [ ] Every release appends one log entry: time, commit, deployment, content, readback result.
• [ ] Approval is a reviewable event, such as a merge with a publish flag.
• [ ] A planted-defect control proves the readback catches a bad deploy.
• [ ] When a blocker clears, record what you observed, mark the cause unknown, and rule out bypasses in writing.
• [ ] Old rules are retired into a dated file, not deleted, so each check's reason stays findable.
• [ ] New rules are added only when a real failure proves they are needed.

Go deeper: release contracts, readbacks and negative controls are covered in the Transition2 course.