FIELD NOTE / INSTAGRAM
Backfill needs a promise ledger.
The complete written thought and the evidence behind it. The video edition will follow its public release.
The written argument is here.
This approved Instagram edition is on the journal now. Its video player and original platform link will appear after each public release is verified.
Backfill needs a promise ledger.
Video caption
Backfill needs a promise ledger.
The job can be late without backfill being the cause.
Start with duration, topology, reservation and faults.
Backfill efficiency is secondary to the sold service.
#EricFieldNotes
Full written post / accessibility read
Backfill can fill idle windows safely relative to the higher-priority jobs it expects. Slurm documents that rule. But the customer was sold a time, not a prediction. Put promised start, predicted start and actual start on one card before calling the scheduling policy successful.
Imagine a reservation promised at noon. The scheduler predicts one o'clock because another job's duration estimate and topology block the slot. A short backfill run uses spare capacity and ends before one. The customer is late, but removing that backfill would not fix the noon promise.
Replay the same queue with the backfill job removed, then vary time limits and placement. Inspect Slurm's reservation trace, predicted start updates and actual allocation. If the backfill truly delayed an expected start, show the evidence; otherwise repair the estimate or the commercial promise.
Keep backfill when it yields useful work without violating the configured expected-start rule. Maintain a separate SLA check against external promises and revise reservation policy on misses. Do this because scheduler correctness and customer success require two proofs.
#EricFieldNotes
Four-beat scene transcript
1. Backfill needs a promise ledger.
Backfill can fill idle windows safely relative to the higher-priority jobs it expects. Slurm documents that rule. But the customer was sold a time, not a prediction. Put promised start, predicted start and actual start on one card before calling the scheduling policy successful.
Visual: The scheduler's prediction is not the customer's deadline.
2. A green utilization chart misses the gap.
Imagine a reservation promised at noon. The scheduler predicts one o'clock because another job's duration estimate and topology block the slot. A short backfill run uses spare capacity and ends before one. The customer is late, but removing that backfill would not fix the noon promise.
Visual: The job can be late without backfill being the cause.
3. Audit each miss causally.
Replay the same queue with the backfill job removed, then vary time limits and placement. Inspect Slurm's reservation trace, predicted start updates and actual allocation. If the backfill truly delayed an expected start, show the evidence; otherwise repair the estimate or the commercial promise.
Visual: Start with duration, topology, reservation and faults.
4. Protect the promise, not the chart.
Keep backfill when it yields useful work without violating the configured expected-start rule. Maintain a separate SLA check against external promises and revise reservation policy on misses. Do this because scheduler correctness and customer success require two proofs.
Visual: Backfill efficiency is secondary to the sold service.
Research and claim limits
The examples identified as illustrative or simulated are design probes, not reported incidents. Vendor specifications do not establish workload performance.