A read-only observer still needs backpressure

The disk auditor could make the disk incident worse without changing a single product file.

I had treated “read-only” as a safety property. It describes one kind of side effect: the observer does not edit the data it measures. It says nothing about the resources it consumes while reading. A recursive du can compete for I/O with the very builds, databases or recovery work an operator is trying to understand. Setting the command to idle I/O priority helps, but it is not admission control on every storage stack. The implementation notes explicitly call out multi-queue devices where ionice idle alone does not guarantee that the scan yields.

The design change was to make observation react to pressure while its most expensive child was still running. An initial host-pressure check is not enough: a scan may start when the machine is quiet and meet an incident halfway through a long traversal. The auditor now passes a stop callback into the du helper. That child runs in its own session, and the helper polls in bounded intervals of at most one second. When the callback sees pressure, it marks the scan partial, signals only that private child process group, waits briefly, and escalates to SIGKILL if needed. It does not signal unrelated work on the host.

while True:
    if should_stop() or time.monotonic() >= end:
        return None
    try:
        out, _ = child.communicate(timeout=min(1, remaining()))
        break
    except subprocess.TimeoutExpired:
        continue

The callback is connected to the scan’s pressure logic. The code records I/O pressure samples, uses a duty-cycle delay after heavy reads, and changes the mode to light if pressure interrupts its owned du. The retained implementation has two related rules: sustained samples at or above 40% I/O PSI some avg10 for three consecutive samples truncate the scan; while a du is active, the stop callback asks whether avg10 exceeds its 25% limit. The thresholds are operational policy in this specific auditor, not universal values for every host or workload.

This arrangement closes a timing gap that a startup-only check leaves open. The scan asks for admission before launching expensive work, then rechecks while the long-running child is alive. If pressure rises, it returns to lightweight inventory rather than continuing to answer a disk question at the cost of making the disk busier.

Partial evidence must stay visibly partial

Stopping du means some paths were not measured in this run. The auditor cannot turn “not observed” into “zero bytes,” and it should not replace a complete last-good inventory with an incomplete document. The code writes a pressure-interrupted result to a separate filename:

destination = a.out + '.partial.json' if 'io-pressure' in doc.get('truncated', []) else a.out
tmp = destination + '.tmp.%d' % os.getpid()
with open(tmp, 'w') as f:
    json.dump(doc, f)
os.replace(tmp, destination)

The temporary file and os.replace make the chosen output atomic at the file level. The partial suffix preserves the normal full output. A consumer that only reads the normal inventory therefore keeps last-good evidence rather than silently receiving the interrupted scan. But now the dashboard has to tell the truth about freshness. Last-good data may be stale. A partial result is useful, but it is not a fresh full scan. The document carries observation time, mode, pressure samples, truncation reasons, item count and duration so a consumer can distinguish “we found nothing” from “we stopped before we finished.”

There is another subtle state interaction: the size cache must not prune entries simply because a pressure-limited traversal did not revisit them. The scanner retains recent seen| markers for rotation and avoids pruning on an io-pressure run. Otherwise a deliberate partial scan would destroy useful prior knowledge about paths it never had the resources to inspect. It still saves the updated caches, but the absence of a fresh observation is not treated as evidence that a cached path disappeared.

This is where “degrade gracefully” often becomes a dangerous phrase. Graceful degradation has to preserve the failure reason. If the observer publishes a smaller inventory under the same path and the consumer has no freshness or completeness check, the system has converted a resource limit into a false healthy answer. The state needs at least three distinctions: last-good complete result with its age, current partial result with its truncation reason, and no usable result. The consuming dashboard or capacity tool must decide which of those states can support a decision.

Priority and backpressure solve different problems

Lowering priority asks the operating system to schedule this work behind other I/O. Backpressure asks whether the observer should continue issuing more work at all. A low-priority scan can still create load, especially if its priority mechanism is ineffective on the underlying device or if enough workers do the same thing together. A pressure callback gives the scanner a way to stop the one expensive traversal it owns and switch to a cheaper mode.

The scope of termination matters. The du helper starts its child in a new session. The stop function signals that child’s group, first with SIGTERM, waits briefly, then escalates to SIGKILL; it does not search the process table and kill every process that happens to be named du. That makes cancellation ownership narrow and auditable. A telemetry utility should never turn a host-wide pressure signal into broad process cleanup.

The policy also should not pretend that one interrupted du is the whole scan. Other inventory stages may still complete. The record marks the scan as light and truncated, and reports what it collected. That is useful operationally: an operator can still see file-system capacity and some metadata while knowing that allocated-size measurements were curtailed. The data’s usefulness depends on carrying its limitations next to the result.

I would review an observer by asking: What does it read, how much can it read at once, how does it know the host is under pressure, can it react during the longest operation, what exact child processes may it stop, and how does it preserve last-good state? Then I would inspect every consumer. If an alert treats a stale full scan as current, or treats a partial result as a clean inventory, the observer’s careful output handling has not reached the decision boundary.

For a safe drill, run the scanner against a disposable tree while applying controlled I/O pressure. Confirm the callback interrupts only its own child group, the scan records io-pressure, the regular complete file remains byte-for-byte untouched, and a separate partial file records time and truncation. Then remove the stop callback as a negative control and prove the test notices that the child continues. Finally, point the real consumer at both files and make it display age and completeness instead of silently choosing the most reassuring interpretation.

This makes observation a resource policy. It can spend the system’s remaining I/O budget while the system is healthy, then give way when the workload needs that capacity. The operator gets less detail during pressure, but the detail that remains is honest about what the scanner did and did not see.

How this was verified

I checked scan.py and aalib.py at gorunner commit efd0bb51fe572dec5674e106808e657b78894e14 against the dossier’s pinned hashes. The source implements pressure checks during an owned du, bounded polling, private process-group TERM/KILL, cache-retention behavior and a separate .partial.json output. This writing pass did not create live I/O pressure, rerun the auditor’s full suite or measure a before/after scan timeline. The thresholds above are code settings for this candidate, not a general recommendation or a measured service outcome.