Restart is where a deletion tool can lose the only backup

A destructive archive tool can pass its first run and still be unsafe to resume. The dangerous moment is the gap between a successful upload and a durable record that the batch completed.

I was reviewing an archive-and-delete workflow before letting it continue through a large log cleanup. The planning scan included conversation records, live task state and package cache data that should not have been treated as disposable logs. We protected those paths before deletion. The scan also created batches, uploaded an encrypted archive, read the object back, checked its digest and members, then removed source files whose identity still matched the plan.

The safety review found a separate restart hazard: the object name for a batch was deterministic, but the completion receipt was appended only after the deletion work. If the upload and source deletion completed but the process stopped before writing that receipt, a restart could treat the batch as unfinished and upload again to the same object key. That retry could replace the only backup. I chose not to restart the live run while that state transition was unresolved.

A successful upload is not a completed batch

The operation has several states: planned, uploaded, read back, verified, source revalidated, deleted, and receipted. A single “done” flag hides where a restart landed. If the process dies after uploading but before validation, the archive may be incomplete. If it dies after validation but before source deletion, the data is safe but still consumes space. If it dies after deletion but before the receipt, the original is gone while the system may still believe the batch is pending. The recovery action must depend on the last durable state, not on a retry that repeats everything.

This is the same reason a payment worker cannot equate “request timed out” with “nothing happened.” External storage and local deletion are separate effects. A crash can happen between them, and the retry must discover which effect already occurred. An idempotency key helps only if the storage operation will not overwrite the verified object. Deterministic naming without conditional creation can make retries destructive.

Make the archive immutable before removing the source

I want the upload to have a unique immutable object identity or a conditional create that refuses to replace an existing batch. Then the tool can read the object back, compare its whole-object digest, and verify every expected member name and size. Only after that should it write a durable “verified” state. A restart that sees that state must continue from source revalidation and cleanup, not upload over the archive.

The state record should be append-only and identify the run, batch, object version, source manifest, digest, and each transition. If the process stops after the verified record but before deletion, recovery can safely retry the delete step. If it stops after deletion, recovery can verify the receipt and mark the batch complete without touching the object. If a batch has no verified archive, the source remains in place and the run blocks for investigation.

A restore exercise is part of this design. Reading back a checksum proves bytes arrived intact; it does not prove that an operator can decrypt, enumerate and restore the intended files. I restored a trial batch manually rather than using the deletion tool’s own recovery path. That demonstrated that some data could be recovered, but it did not verify the tool’s restart behavior. Recovery needs to be rehearsed through the same interfaces that will be used during an incident.

Revalidate the world at deletion time

The plan is a snapshot, not permanent authority to delete. Before unlinking, the tool should check that each file still has the planned device, inode, size and modification time, that no process has it open, and that it is still outside every protected category. The current archive code checks file identity and open handles and reapplies the current protected-prefix list. But version-control tracking can change after planning; a path that was untracked when the plan was written could become tracked before apply. The delete phase should recheck tracked status against the current repository state too.

This revalidation protects against a changing filesystem and evolving policy. It also gives a clear decision for uncertain cases: leave the file alone and report why. A destructive cleanup should prefer a missed deletion over deleting a path whose status changed underneath the plan. The run can be resumed after review, with a new plan for changed inputs.

Treat interruption as a normal state

I design batch tools assuming they will be interrupted. Process restarts, network timeouts, worker eviction and operator stop requests are ordinary operating conditions, not rare exceptions. For each stage, I ask: what durable evidence exists, what can safely repeat, and which side effect might already have happened? Those answers define the state machine and its recovery tests.

For archive deletion, the essential properties are: never overwrite an existing verified backup; never delete before readback and member verification; never trust a stale plan after relevant file or policy changes; and never report completion without a durable receipt. A crash injected after every transition should leave either the source intact or a verified immutable archive that the resume path recognizes.

The lesson applies to migrations, deployments, backups and any workflow that combines an external write with a local destructive action. Correctness on a clean run is not enough. The restart path is part of the product contract. What would your job do if it stopped one instruction after the irreversible effect?

Make retry identity different from archive identity

A retry-safe destructive workflow needs to distinguish “the same planned batch” from “the same upload attempt.” If both map to a single deterministic object key, an incomplete retry can replace the only verified archive. One option is an immutable attempt identifier for every upload and a durable manifest that points to the verified attempt. Another is a storage precondition that refuses replacement. Whichever design you choose, a process restart must discover whether an object exists, read it back, verify its digest and membership, and resume from that recorded state rather than repeating an upload blindly.

The state record should be written ahead of destructive work and updated after each durable transition. A crash between two transitions should leave an unambiguous state to reconcile. Before deletion, revalidate each source against the plan so a file modified after the scan is preserved. After deleting, record which exact members were removed. A “batch complete” boolean is too weak if it cannot explain whether upload, readback, verification and deletion each happened.

Design the restart matrix before the delete call

I treat every side effect as a crash point. Before upload, during upload, after the object is stored, during verification, after source deletion and before the final receipt are separate restart states. For each one, I ask what durable facts exist and what retry is allowed to do. If retry can replace an already verified archive, the operation is not idempotent even if the happy path usually succeeds.

The archive should have a stable identity derived from the source object and the operation, with create-if-absent semantics. If the object already exists, the tool reads it back and compares its identity and content before deciding whether to continue. A mismatch is a conflict that needs operator attention; it is not permission to overwrite. After verification, the source is revalidated immediately before deletion so a changed source cannot be mistaken for the one that was archived.

I also make the resume record useful to a human. It should show the last completed state, source identity, archive identity and next safe action, while excluding credentials. Then an interruption drill can stop the process at each boundary and prove the next invocation either resumes safely or refuses. The test is about restart behavior, not merely successful deletion.

I also keep a recovery path that does not depend on the same worker that may have crashed: an operator can inspect the journal, verify the stored artifact independently and decide whether to resume or stop. What would your deletion tool do if it restarted after the upload succeeded but before its completion receipt was durable?

How this was verified

I checked the archive tool’s current upload, readback, revalidation and delete order against the captured review finding. The source digest is pinned in the packet. This piece omits assistant-reported file counts and storage totals that were not tied here to a durable, privacy-safe receipt.