Design recovery semantics into the safety state machine

At 02:48:06 UTC, our pressure monitor restarted and reported CRIT. Its desired level was GREEN. The historical blocked-task evidence was still present. Sixty-five seconds later, the reported level was WARN, and subsequent readbacks showed real worker turns advancing.

That sequence was the outcome I wanted. I needed a gate that could recognize recovery without forgetting why it had stopped work. Clearing the history would have removed useful evidence. Treating every retained warning as a newly active failure would have kept the system unavailable after the underlying condition had resolved.

This is an operations problem with a precise implementation consequence: diagnostic retention and present authority to block work need different state. I built the recovery path around that distinction, including process identity, startup replay, bounded uncertainty and the existing cooldown. The useful artifact is a contract for when the gate may relax, plus evidence that downstream work actually resumes.

Separate the diagnostic record from admission authority

The original kernel warning said a task had been blocked for more than a threshold. That observation was real. Keeping it for thirty minutes was useful for diagnosis. Sampling the monitor every five seconds did not make that old kernel event a new warning every five seconds.

I separated three facts in the event output. blocked_history says a retained warning exists. blocked says current evidence still authorizes the critical hold. history_incomplete says the monitor cannot confidently reconstruct the relevant history. The latter is independently critical because an incomplete archive is not proof that nothing dangerous happened.

A dashboard can show all three. The admission policy needs to distinguish their meanings. If a retained historical flag directly renews critical authority on every sample, collection freshness hides event age. The JSON file looks current while the decisive observation becomes older. This is how a well-intentioned retention policy can become an accidental availability policy.

I kept the original event timestamp attached to each warning. A collector timestamp can answer when we read the record; it cannot substitute for when the kernel observed the condition. The same distinction matters in circuit breakers, incident aggregators, fraud queues and dependency-health monitors. Repeated collection is not repeated occurrence.

Give fresh evidence its full conservative interval

The recovery rule did not immediately dismiss a warning because its named process disappeared. A fresh blocked warning retains its hold through the existing ninety-second boundary. There may be other effects of the episode that the next sample has not yet captured. I preserved that conservative interval rather than using a single favorable observation to jump directly to normal service.

Beyond the interval, the code asks a more specific question: can current process evidence establish that this warning no longer describes an active blocked task? A known absent PID can resolve that warning, provided the proc root itself is readable. A matching process incarnation in a recognized recovered state can also resolve it. Unknown identity cannot.

Here is the actual decision function, retained verbatim from the reviewed source:

    def _blocked_active(self, event, now):
        if now - event["at"] <= STALE_SECONDS or event.get("blocked_ambiguous"):
            return True
        pid = event.get("blocked_pid")
        if not pid:
            return True
        identity = self._blocked_identity(pid)
        if identity == "gone":
            return False
        if not isinstance(identity, tuple) or identity[2] != event.get("blocked_start"):
            return True  # Unreadable, malformed, unobserved or reused PID.
        return identity[1] not in {"R", "S", "I", "Z", "X", "x", "T", "t", "K", "W", "P"}

The function is one part of the monitor, not a portable declaration that every resource problem is solved when a process sleeps. Other critical signals still evaluate independently. A recovered warning cannot cancel a Docker timeout, a fresh blocked task or another critical pressure measurement.

A process number is not a process incarnation

PID reuse makes naive recovery surprisingly dangerous. A warning can name process 17. Later, process 17 may be sleeping normally because the operating system assigned the number to something else. Seeing a healthy process with that number does not prove the original task recovered.

The implementation records the process start value when identity can be established. It compares that incarnation when reevaluating the old warning. A different start value remains ambiguous and keeps the warning active. It also checks the recorded command identity and relates process birth to the original event time. A process born after the warning cannot be its original subject.

The distinction between absence and unavailable evidence matters just as much. A missing stat file under an existing process directory is not established process exit. An unreadable or malformed file is not a healthy process. Even the claim that a PID is gone requires a readable proc root and an absent PID directory. Otherwise an unavailable observation mechanism could accidentally become permission to resume.

This is an identity pattern worth reusing beyond Linux. A worker name needs its lease generation. A deployment name needs its revision. A connection needs its session identity. Recovery should refer to the entity that produced the original condition, rather than a convenient label that can be reused.

Preserve state when the observer restarts

Restarting the observer is itself a state transition. Starting from GREEN would discard the hold that existed before the restart. Re-reading a recent warning and assigning the current time would create the opposite defect: the warning would regain a fresh interval merely because the observer restarted.

I restored the saved level and performed one bounded kernel-journal backfill before the first normal tick. The backfill retains the journal timestamps. The two existing live followers remain responsible for new entries; this did not add a process on every sampling cycle or create another continuously running service.

Startup failures have their own semantics. An invalid saved level, malformed history, future timestamp, oversized result or potentially truncated result establishes uncertainty. The implementation keeps that authority separately for thirty minutes. The recovery contract therefore includes a bounded uncertainty period, rather than an indefinite latch that requires someone to delete a file.

The bounded period is a policy choice with trade-offs. It does not manufacture missing evidence. During that period, other current signals still matter. When it expires, the monitor reevaluates under the remaining evidence and normal hysteresis. A system with different consequences may need a different owner or reconciliation action. The essential design step is to specify the uncertain state and its exit path explicitly.

A bounded buffer needs a policy for lost evidence

The event deque holds at most 2,048 entries. The startup command also bounds its requested journal rows. Those limits protect the monitor from unbounded memory and replay work, but they create a correctness question: what does a full result mean?

Exactly 2,048 returned history rows might mean the window contained exactly that many events. It might also mean the query omitted older relevant entries. The monitor cannot tell from the count alone. The implementation treats that exact-cap result as potentially truncated and holds uncertainty conservatively. An eight-megabyte response bound and a five-second command timeout constrain the replay through different dimensions.

The harder issue was concurrency. The live journal followers can append while startup history is being prepared. Checking capacity before taking the event lock would leave a gap between the check and insertion. Even a history batch that fits on its own could overflow the deque when combined with newly arrived live entries.

The final implementation checks the combined live-plus-backfill capacity and extends the deque under the same lock. On overflow, it preserves the existing live events. It does not append a synthetic warning into the already full deque, because that marker could evict the real evidence it is supposed to protect.

Instead, incomplete-history authority lives outside that deque. Busy ordinary follower traffic can evict retained events under the normal bounded policy, but it cannot evict the separate uncertainty hold. Memory bounds are part of the state machine; they need an explicit answer for what happens when evidence no longer fits.

Recovery needs hysteresis and an observable postcondition

When desired severity becomes lower than current severity, the monitor retains the existing sixty-second per-level cooldown. Recovery moves one level at a time. CRIT does not jump directly to GREEN because one sample appears healthy. If critical conditions return, the below-threshold recovery interval starts again.

This distinction gives operators two useful values: desired level from current signals and effective level after hysteresis. At the first recorded activation sample, desired GREEN and effective CRIT were compatible. The monitor was observing recovery while preserving the prior hold. At the later sample, WARN was compatible with retained historical warnings and no active blocked or incomplete-history signal.

I also needed a downstream postcondition. A gate can accept queued work without workers making progress, and a status badge can change while another queue defect remains. The activation record therefore includes subsequent worker and review turn counters. One foundation task advanced from thirteen to eighteen turns; a harness task from zero to four. Review counters advanced from seventy and sixty-six to seventy-eight and seventy.

Those observations establish resumed inference for those tasks. They do not establish completed customer features, healthy external providers or a successful product release. Keeping the postcondition narrow makes it more useful: it identifies exactly which layer recovered and which acceptance work still belongs elsewhere.

Test the transition and the evidence that can invalidate it

The original review covered fresh warnings, the ninety-second boundary, live blocked state, PID reuse, malformed identity, original timestamps, saved CRIT and the sixty-second drop. It also found a concrete startup race: two existing live events followed by 2,047 historical ordinary records could evict a fresh blocked warning in the earlier candidate.

The corrected implementation preserved live evidence and established the independent uncertainty hold. The historical reviewer killed five separately chosen source mutations, including removal of the freshness hold, bypassing incarnation comparison and ignoring exact-cap truncation. The previous candidate failed the new race probe; the final reviewed source passed it.

For this article package I retained the whole source and its original test fixtures, then added a compact offline exercise using synthetic proc directories, an injected clock and scripted journal responses. It checks the recovered state, retained history and boundary behavior rather than asserting that a function name appears in the file. No live kernel history or service restart is needed to run that exercise.

The useful negative control removes a guard or changes a boundary and reruns the behavior. A test should fail because the observable recovery contract changed. A syntax error or missing fixture is an infrastructure failure, not a detected safety defect. I record those separately so the evidence does not become stronger merely because the test environment is incomplete.

Design the operator action as carefully as the classifier

A correct classifier can still be activated badly. The historical handoff constrained activation to the existing service, existing configuration and existing state location. The reviewed file hash was checked before restart. The service unit and retained history were preserved. No cap increase, broad installer run or cleanup of evidence was part of recovery.

That operational boundary matters because a broad repair can remove the very state that would expose a regression. If the first sample starts healthy only because someone deleted the prior hold, it does not demonstrate that startup recovery works. The expected first sample was therefore conservative: retain CRIT, retain history, then observe the existing cooldown and downstream work.

This is also a useful division of ownership. The code author can prove offline transitions and provide an activation contract. The operator who activates the service owns source readback, first-sample evidence and actual downstream observation. Neither should quietly substitute their evidence for the other's. A replay is not an activation; an active service is not proof that its recovery logic preserves the required invariants.

Compare recovery policies before choosing the convenient one

There were several plausible alternatives, and each predicts a different failure signature. Keeping every warning active for its full retention period is simple and conservative, but it makes recovery latency depend on archive duration rather than current condition. Deleting history restores availability quickly, but loses both diagnostic evidence and protection against a genuinely unresolved episode. Clearing on absent PID alone is cheap, but confuses process exit with an unreadable observation mechanism and ignores reused identities.

The chosen policy spends more complexity on evidence interpretation. It preserves the fresh interval, proves absence or a matching recovered incarnation, holds ambiguity, then applies normal per-level hysteresis. That complexity is justified only if the tests can discriminate those alternatives. The synthetic cases therefore include an absent process under a readable proc root, an existing process with missing stat data, a sleeping reused PID and a same-incarnation sleeping process. These scenarios look similar in a summary and must produce different decisions.

There is also a deliberate difference between bounding uncertainty and declaring it resolved. Expiry stops an old incomplete-history marker from blocking forever; it does not reconstruct the missing records. Operators still need the retained uncertainty fact, its age and the current signals when deciding whether further investigation is warranted. A higher-consequence system could require an explicit reconciliation receipt instead of time alone. That would be a different policy with a named owner and an availability budget.

I would put those alternatives and predicted failure signatures into the design record before changing a timer. A shorter timer can make a demo look responsive while weakening the evidence contract. A longer timer can hide an identity bug by making the system appear cautiously unavailable. The decisive experiment must exercise the identity and transition semantics, not merely wait until a badge changes color.

Make availability a specified safety outcome

A safety gate should describe how it stops work and how it becomes safe to resume. Otherwise, the most reliable way to restore service eventually becomes an undocumented override. That is a weak place to put both availability and trust.

My checklist starts with the observation: event time, subject identity and evidence completeness. Next comes authority: which facts currently require a hold, and which facts are retained for diagnosis? Then comes transition policy: freshness, uncertainty lifetime, hysteresis and the conditions that restart the recovery interval. Finally comes the observable postcondition in the system behind the gate.

For a circuit breaker, that may be successful bounded trial requests. For a deployment gate, it may be the exact revision serving the expected behavior. For a worker admission system, it includes accepted work actually advancing. The postcondition should match the layer being repaired rather than borrowing a larger business outcome.

The design question I now ask during review is straightforward: after the original cause resolves, what precise evidence permits this system to recover, and can we observe that recovery without deleting its history? If the answer requires an operator to remember which files to remove, the recovery contract is unfinished.

How this was verified

The source packet pins the complete reviewed StormGuard source by Git object and SHA256, together with the historical recovery note, independent review and original offline fixtures. The 19 October activation timestamps and worker counters above come from that committed operator receipt, not a new activation performed for this article. Historical review results remain historical.

The article package separately records its current offline owning-suite execution, compact synthetic recovery exercise, executable negative controls and independent review of that exercise. It does not contact providers, restart services or assert a new production recovery. CRIT-to-WARN, resumed worker turns and product acceptance remain distinct outcomes. The diagram explains this implementation's contract; it does not claim a universal recovery policy for every pressure source.