Distributed execution and policy / Field essay 18

Regional cells need one home for mutable authority.

Portable containers do not make state ownership, residency and failover portable. Define one authoritative home for mutable state before adding regions.

Published

ARCHITECTURE
FIELD ESSAY 18 / DISTRIBUTED EXECUTION AND POLICY

Regional cells need one home for mutable authority

A second region gives you another place to run code. It also forces a decision about who may change the truth.

That became the organizing question in my regional architecture for OptinRouter. I wanted portable gateways and workers without turning every place we could run a container into another writer of account, money and consent state.

The design gives an organization a home cell. Its mutable records have a known write authority, while approved ingress and compute can live elsewhere. A cell contains the regional writer, queues, keys, policy configuration and artifact responsibilities needed to operate that authority.

This is an architecture direction, with a small decision model I wrote to make its acceptance conditions executable. It is not a report of an activated multi-region deployment. The useful question is how to grow from one operational unit to two while keeping ownership, permissions and recovery coherent.

Start with the obligation, not the topology

When I plan the first deployment, I separate a synthetic demonstration, an operator-funded pilot and a system holding customer-funded obligations.

Those uses can share application code while carrying different recovery consequences. Losing a disposable demonstration database is a different event from losing the sole record of a customer's balance or a supplier liability. Calling both deployments a pilot hides the decision that matters.

My proposal allows a deliberately limited one-VM demonstration or operator-funded profile. The default paid profile gives the database durability independent of the application VM. A one-VM paid exception needs a named accountable owner, explicit exposure ceilings, a loss policy and an expiry or migration gate.

Those are operating decisions. I would want to know the maximum customer funds at risk, maximum supplier exposure, maximum unreconciled obligations and the trigger that stops new admission. Backups remain necessary, but the existence of a backup does not answer how current permissions and obligations survive restoration.

That framing changes a CTO discussion. Instead of asking whether the platform is sophisticated enough, I ask which failure we are accepting, who owns the consequence and what evidence would justify the next deployment profile.

Give each mutable object one authoritative home

The regional-cell design assigns mutable account, financial, consent and submission state to an organization's home authority.

Stateless ingress can authenticate a request and resolve the home cell. An approved worker can perform bounded computation. Neither should acquire independent write authority over the same wallet or consent record merely because it runs in another cloud.

The small model makes this distinction visible:

    if intent.executing_cell != authority.home_cell:
        return 'route_to_home'
    return 'allow_home_write'

Those lines come from the illustrative model, after its other eligibility checks. They describe a decision, not a network proxy or a database fence. Routing an intent home still requires an authenticated service path and current authority at the committing sink.

The architecture needs that separation because compute placement and ownership evolve at different rates. I may want another worker location for capacity or a recovery target before I am ready to operate another writer. Treating those as separate changes lets the system gain useful options without inventing a global conflict-resolution problem for money or permission history.

One home per mutable object also gives incidents an address. The operator can identify the authority that decides whether a write is current, rather than asking which of several apparently healthy copies should win.

Keep the directory smaller than the data plane

A global directory should carry the minimum approved information needed to find a tenant's home cell. It should not become a convenient global copy of every customer's content and mutable state.

The request path authenticates and resolves the tenant before making the routing decision. A user's current IP geography does not rewrite the tenant's residency policy or move its account authority.

I would version and audit home-cell changes because the directory participates in ownership transfer. A directory update is consequential when it can send new writes toward a different authority. It cannot be treated as an ordinary cache refresh while both old and new cells continue accepting changes.

The directory also needs an uncertainty answer. If the current home mapping cannot be established under the required policy, sending a request to whichever region responds fastest would substitute availability for authority. The system needs a bounded refusal or hold with an operational owner.

That does not require every read to become globally synchronous. It requires naming which directory facts are advisory and which are necessary before a protected operation may proceed. The write path should not inherit an optimistic assumption from a latency-oriented ingress layer.

Separate placement from processing evidence

A configured region is a claim about intended placement. A processing-location observation answers a different question.

The enterprise design makes region eligibility explicit for the main inference path and for side actions. The decision model therefore requires both a verified location fact and membership in the policy's allowed-region set. A syntactically valid region name cannot establish that the selected provider or adapter actually processed there.

This is broader than the model endpoint. Ingress, TLS termination, request inspection, logs, crash reports, backups, support access and fallback routes can all become part of the approved data path. A regional origin does not erase earlier processing elsewhere in the request's journey.

For a deployment review, I would draw that path before choosing the label for the product. Which component receives private content first? Which side action can copy it? Which artifact or diagnostic record leaves the cell? What evidence establishes each destination's eligibility?

The model does not discover any of those facts. It consumes a prevalidated location fact to illustrate the admission rule. The production adapter must supply the evidence and preserve the restriction through actual dispatch, redirects and fallback.

A policy snapshot needs a deadline

A gateway can retain a policy snapshot during a control-plane interruption. That helps only while the snapshot remains within the authority's permitted freshness interval.

My runtime proposal sets a maximum safety-sensitive permission age of 30 seconds, with a proposed five-second same-cell polling interval to qualify. Those are design choices, not measured deployment behavior. The maximum age expresses a revocation requirement; it should not silently expand because a busy system would prefer to keep admitting work.

In the teaching model, the interval is explicit: issued time is included, and the earlier of declared expiry and issued time plus 30 seconds is excluded. The exercise accepts a snapshot just before that boundary and holds at the boundary itself. This half-open convention is a model choice that makes the edge unambiguous.

I also test a snapshot whose declared expiry is far in the future. It still cannot bypass the 30-second ceiling. Otherwise a field intended to support a shorter validity could accidentally authorize a much longer one.

If the business requires faster revocation, it needs a shorter lease or a current check at admission. If the adapter cannot meet that requirement, the decision belongs in the operating contract. Increasing the cache lifetime without revisiting the accepted risk changes what the system promises.

An epoch needs an authority that survives the failure

An epoch is useful when it distinguishes an old writer or old permission state from the authority that is current now.

A counter restored from the same lost database cannot, by itself, establish that distinction. It can be internally consistent with the restored snapshot while remaining behind a revocation or ownership transfer that happened after the backup.

That is why my restore design requires a surviving authority for the recovery epoch. The model treats its trusted-epoch input as an externally established fact. Setting that field inside the restored database would not satisfy the real architecture, even if the pure function returns an allow decision when fed it.

The write intent and policy snapshot must agree with the current authority's epoch. In the exercise, an old intent remains held after the home cell and epoch change. A matching new intent at the old cell routes to the new home. Matching the new home, epoch and other eligibility conditions produces the model's allowed-write decision.

This is a way to make the required information concrete. It does not fence an old database connection or prove that an isolated former primary has stopped. Those effects need enforcement at the actual writer and infrastructure boundary.

Restoring data must not resurrect permission

A backup can contain permissions that were valid when it was taken and withdrawn later. Replaying that snapshot as current authority can reopen access precisely when the operator is trying to recover safely.

The proposal therefore keeps restored effectful work behind an authorization barrier. Recovery needs a surviving complete restriction history or fresh authorization under the new recovery authority. Restoring tables and restarting workers is only part of that operation.

I modeled two explicit witness forms to show the difference. A complete-restrictions witness represents an externally verified basis for current restrictions at the accepted epoch. A fresh-grant witness applies to one principal and one operation. It cannot become a global statement that every old grant is valid again.

The exercise changes the witness's organization, epoch, expiry, mode, principal and operation independently. Each mismatch keeps the restored work held. It also checks that a fresh witness cannot override the current policy's write denial or repair an untrusted external epoch.

The model does not prove that a restriction history is complete or that a new grant was genuinely issued. Those are adapter responsibilities with independent evidence. The point is to keep their outputs explicit and scoped enough that recovery cannot quietly replace them with “the database is healthy.”

Availability has several different meanings

A cell can have healthy compute while its write authority is unavailable. Another cell can have spare capacity while remaining ineligible to accept the affected writes. A provider can answer quickly from a location the tenant's policy does not permit.

I want those states visible because they lead to different operational choices.

If the application host fails while durable state remains available, replacing compute may be the immediate recovery step. If the regional writer fails, promotion needs the supported writer procedure and old-writer exclusion. If the authority state is uncertain after restoration, new protected work stays restricted until that uncertainty is resolved.

Adding a second gateway helps one failure pattern. Adding an independently durable database helps another. Creating a second region adds a different set of dependencies and an ownership-transfer problem. Calling all three high availability makes the discussion less precise.

I would review the failure domain alongside the customer-facing obligation. Which requests can continue? Which must wait? What state can be read without implying it may still authorize a new effect? Who decides the unresolved case, and how does the system stop the waiting state from disappearing into a green dashboard?

Migration is a workflow with a closing condition

Moving a tenant between cells changes the authority for its mutable records. I design that as a controlled workflow rather than two writable copies followed by reconciliation.

The sequence starts by quiescing affected writes and resolving active holds and workflows. Allowed data moves with manifest and version checks. Ownership transfers under a new epoch, the directory changes, and acceptance verifies the new path. The rollback plan remains attached to the transfer.

The closing condition matters. A successful copy says the bytes arrived. It does not establish that the old authority stopped accepting writes, that pending obligations were reconciled or that the directory and committing sink agree on the new owner.

For money and consent, I would avoid treating last-write-wins as the business reconciliation policy. A later timestamp does not explain which permission was current or whether a balance reflects the same transfer obligation twice.

A cross-cell balance transfer also needs its own operation identity and pending, completed or reversed states. When the other side is unreachable, unresolved work remains pending and reconciles. Neither side should manufacture a spendable result from the assumption that the other side probably finished.

Choose the second location's first responsibility

My first multi-cloud expansion would usually give the second location a deliberately limited job: an approved worker role or a tested recovery target.

That can produce useful evidence before there are two live writers for the same domain. A worker role qualifies identity, data projection, deadlines, output fencing and shutdown behavior. A recovery target qualifies backup inventory, restoration, authorization barriers and operator steps.

Both require implementation and drills. Neither automatically grants active-active write authority. The value is that the next investment answers a specific operational question instead of creating a topology whose meaning the team has to discover during an incident.

This also improves vendor choices. I can compare whether an environment supports the role's actual contract rather than treating container compatibility as proof that networking, identity, draining, keys and recovery behave identically.

The portable artifact is the image and application protocol. The deployment adapter still owns the environment-specific obligations. I would rather certify a small supported set of roles honestly than let a broad compatibility claim hide an untested authority boundary.

Measure the recovery path separately

A deployment diagram needs a failure matrix beside it. For each affected domain, I would record the current owner, trusted epoch source, permitted recovery target, write-stop trigger, pending-operation treatment and the evidence required before resuming.

The measurements should follow those obligations. Time to start a replacement process is different from time to restore current permission authority. Time to copy data is different from time to reconcile outstanding supplier observations. A successful backup job is different from a measured restoration and acceptance drill.

The model's case count is not any of those operational measurements. It tells me the example distinguishes the selected admission decisions. Production recovery objectives still need their own observed start, completion and acceptance boundaries.

For a first two-cell drill, I would retain the original operation identities, deliberately make the old writer unavailable and verify that new writes cannot proceed through stale authority. I would then restore into the recovery target, establish the surviving epoch and restriction basis, and verify the current path before workers resume protected effects.

Every step needs a named owner and a stop condition. A recovery plan that depends on someone deciding what to trust in the middle of the incident is missing part of its design.

Write the authority map before buying another region

The decision tool I use is a small table: mutable domain, authoritative home, current epoch source, permitted compute locations, freshness requirement and recovery barrier.

A wallet may share a deployment with a consent service while requiring a different reconciliation check. A global directory may be useful without carrying private content. A remote worker may produce an artifact without being allowed to publish a new authoritative record.

Choose one real operation, such as a paid request or a consent change, and trace it through that table. Then ask what happens if the home cell is unreachable, the snapshot expires, the old process returns or the backup predates a revocation.

Those answers determine whether a second region is useful capacity, a recovery target or a new write authority. Making that choice explicit is the part of regional architecture I would settle before allowing the topology to make it for me.

How this was verified

The source packet retains the runtime and enterprise design at OptinRouter commit 88eb34a7d635ce8076b8a06ca90d5e451ce4b78b. Both are architecture direction, not deployed-cell evidence. The accompanying decision model is new educational code derived from those documents; it is not a verbatim production guard.

A local 61-case matrix exercises its typed, prevalidated inputs: tenant and epoch agreement, trusted epoch requirement, half-open snapshot freshness, write denial, scoped restore witnesses, verified-region eligibility and home-cell routing. Three author mutations remove the restore barrier, home routing or fresh-grant scope checks; each fails the exercise, and the restored model passes. Independent review remains pending. No signatures, actual geography, grant history, writer fencing, database writes, network routing, regional migration or production recovery objective is qualified by that matrix.

Independent review was pinned before the exercise: the initial 61-case version caught 28 of 29 mutations with a clean control. The survivor changed the diagnostic when policy was both stale and denied. I added that combined case, preserving stale-policy precedence. The 62-case follow-up caught the same 29 of 29 with control exit zero; this is a same-set follow-up, not a fresh independent score. The model source and locked narration stayed unchanged. These results qualify the teaching model only, not a deployed regional security boundary.

CONNECTED READINGForward and back
Keep reading

The answer deserves
another experiment.

Back to the journal