The offline test that spent the production quota
A test I believed was offline reached a live publishing API and used up the remaining daily request budget. The broken assumption was not just “the fake failed.” We had two separate holes: one path still had network access, and an earlier negative control found no input, so it passed without exercising the behavior we meant to test.
A fake is not a network boundary
A fake transport can make one unit test deterministic, but it does not prove that every route in the process uses the fake. A second module may hold its own client reference. A helper can call a public function that imports the real transport. A copied test can point at an empty render directory and never reach either client. The test can report green while production state changes outside its assertions.
The incident made the shared nature of a quota concrete. Test traffic, scheduled posts and the site’s scheduled refresh all draw from one provider budget. A request that creates nothing can still consume capacity. A malformed payload can still count. Production availability depends on isolation at the process boundary, not on the test author’s intention.
I stopped treating “offline” as a fixture setting. The test runner now blocks real network access for the suite, and the publishing entry points fail closed when the run is marked offline. That gives us two defenses: the application should choose its fake, and the environment should make an accidental real request impossible. If either layer is removed, the other still has to stop the call.
A negative control must actually do work
The earlier control looked reassuring: it passed without sending a request. The reason was a path bug. The copied script looked for rendered files relative to its own directory, found none, and skipped the loop. It proved only that an empty input produces no output.
I now require controls to establish their own preconditions. Before the test begins, assert that the fixture directory contains the expected input. During the test, assert the fake received the intended call. For a guard, remove or mutate the guard in a controlled copy and confirm the forbidden behavior becomes observable. An empty result can be a correct answer, but it is not evidence unless the test proves it saw the case it was meant to evaluate.
This is especially important for integration boundaries. If a test says “the provider client was not called,” it should also prove the request reached the decision point where the client would have been called. Otherwise the guard may be bypassed by dead code, a fixture mismatch or a bad working directory.
Detect effects the ledger cannot explain
The second control is a readback against what the scheduler believes it created. We compare the provider’s scheduled objects with our local ledger and report any object that has no corresponding ledger row. The positive control plants a fake unknown object and confirms it is found. The empty control has no planted object and confirms there are no false positives.
That detector does not prove a live write never happened. It gives us a way to identify state that our normal write-ahead record cannot explain. The ledger is written before the API request so a crash leaves a pending row. A timeout remains ambiguous until a readback reconciles it. We never turn an unknown outcome into a blind retry, because the first request may have succeeded even if the response did not arrive.
The test for this detector has to pin its own data source and make a planted unknown entry visible. Otherwise it can repeat the same empty-control failure in another form. We verify the path with both “one unexpected item exists” and “none exist,” using the fake provider so the proof itself cannot consume production quota.
Give quotas an owner and a budget
External quotas are shared production resources whether or not the code lives in a production directory. A local test can exhaust a daily allowance, delay an hourly refresh or prevent a real operator action. Treat those limits like a database or queue: name the owner, track the remaining budget and make tests unable to spend it.
The practical controls are straightforward. Deny network egress in offline test jobs. Make live mode require an explicit environment and credential source that CI does not inherit. Fail before the first request if offline mode is active. Record intent before a consequential write. Reconcile unknown outcomes by readback rather than retry. And make the test prove the network deny works with a deliberate outbound canary that must be blocked.
One more thing changed in how I evaluate a “green” control: I ask what input it consumed and what observable effect it would have produced if the protection were absent. If the answer is “none,” the pass is vacuous. What shared external quota does your test suite still have permission to spend?
Build an offline test boundary that fails loudly
I want three layers to agree. The application should select a fake transport in test mode. The test runner should deny external network access. The test should make a harmless outbound attempt and assert it is rejected. If the fake is bypassed, egress denial should stop the request; if egress is accidentally opened, the test should fail because its canary reached a destination. This turns “offline” from a convention into a property the job can demonstrate.
The fixture also needs to prove it loaded. Assert the expected render or request records exist before calling the scheduler. Assert the fake received exactly the expected operations. Then run a deliberate negative control with the duplicate guard or offline gate removed in an isolated copy and confirm the behavior changes. A control with an empty directory can be green while testing nothing.
For live integrations, separate credentials from ordinary CI and require a named, explicit live mode. Place the daily quota behind a budget check owned by one scheduler rather than letting every test, refresh and operator call spend independently. Persist intent before writes and use readback to resolve timeouts. What canary would prove that a supposedly offline test environment cannot reach your production service?
Prove the test cannot reach the quota
A test double is a code substitution, not a network boundary. The real client may be constructed in a helper, a retry path may use a different transport, or a subprocess may inherit credentials and reach the same service directly. The runner I would trust separates the protection into layers: no production credential in the test environment, egress denied by the runner or namespace, and an explicit client override for the route under test. If one layer is accidentally removed, another still contains the mistake.
The negative control matters. A harmless probe should demonstrate that outbound access is actually blocked; otherwise an offline test may pass because it had no input or never exercised the client. I want the harness to fail when the probe can connect, and the test result to identify which endpoint was denied without leaking a token. Tests that need a remote dependency should use a dedicated sandbox account with a small explicit budget, not production state.
After execution, inspect the request ledger and the service-side quota signal. A green test process does not prove zero external effects. This is the same principle as payment idempotency: verify what the remote boundary recorded, and preserve a clear UNKNOWN result if readback is unavailable. The target is structural: routine tests should have no route or authority to spend production quota, rather than relying on the fact that they did not happen to spend it today.
How this was verified
I checked the committed unknown-post readback and the incident notes in the source packet. I omit the unpinned request count and any claim about the unreported post-reset measurement.