Agent fleets need explicit ownership transfers before more workers

A process can hand another process a file lock without opening a gap in ownership.

That is the mechanism I want at the center of our local agent-admission story. The launcher, dispatcher and driver do different jobs, but a lane should not become available to a competing owner just because responsibility is moving between them. Passing the already-held descriptor over a Unix socket gives the next process a reference to the same open file description. The sender can close its copy while the transferred copy keeps the lock held.

This is a useful architectural unit for agent fleets: an explicit transfer of an existing right to do work. The model's description of its intentions plays no part in that transfer.

Our host admission dispatcher combines that local lock with a queued row, an admitted-start token and a bounded claim interval. I exercised those boundaries using the extracted production definitions, real local file descriptors, kernel locks, a Unix socket pair and SQLite. There are no model calls or worker launches in that exercise. The subject is the ownership mechanism that must work regardless of which model eventually gets the task.

A fleet has several different permission questions

When I ask whether an agent can start, I separate four questions.

Resource admission asks whether this host should take more work now. Model quota asks whether the selected engine or provider has capacity under its own limits. Task ownership asks which process currently owns this attempt. Release authority asks what can move from a completed attempt into a customer-facing system.

A positive answer to one does not supply the others. A free model slot does not identify the owner of a task. A running worker does not establish that its result passed review. A queue admission does not authorize a production release.

This separation gives the operator a much better diagnosis. “The task is waiting” can mean resource pressure, a model limit, an unresolved owner or an acceptance decision. Those states need different responses. Restarting workers for all of them merely adds more activity around the same unresolved condition.

I use the admission implementation here to explain the local ownership layer. The resource scheduler, engine quotas and final release path remain separate systems with their own acceptance evidence. That is also the organizational lesson: name the decision and its owner before deciding how much parallel capacity to buy.

Let waiting work be data

The admission design replaces a process waiting for each queued lane with a dispatcher holding rows. The launcher still performs its initial preparation and placement work. Once it hands the lane over, waiting no longer requires keeping that launcher alive to reconsider the same gates repeatedly.

The row carries the original queue start, priority, placement, arguments and stage. A run-queue entry remains available as a status view so adjacent tooling can recognize that the dispatcher owns the waiting work. That view has the dispatcher's PID and heartbeat.

I care about preserving those interfaces during a change in scheduling architecture. A new queue is much harder to adopt if every board, liveness probe and stop path needs a new interpretation at the same time. Keeping the established status view makes it possible to change who does the waiting while preserving how operators find it.

The module's historical notes describe the earlier waiter population and overhead. I do not use those comments as a fresh performance benchmark. The useful design point survives without a headline about CPU savings: represent waiting work durably, evaluate shared admission conditions centrally, and retain a concrete ownership handoff when a worker is allowed to proceed.

Centralization also creates an operator responsibility. One dispatcher becomes a component whose restart and recovery behavior matter. The rest of the design has to make that trade visible rather than assuming a smaller process count automatically means a more reliable system.

Transfer the held descriptor

The transport helper sends JSON and, when present, descriptors using socket.send_fds. The receiving side uses socket.recv_fds. The important object being transferred is the descriptor for the lane's already-held lock file.

In the exercise, I take the actual lock, send the descriptor through a real Unix socket pair, receive the transferred descriptor and close the sender's original copy. A fresh attempt to take the lane lock still fails. Once the received descriptor is closed, the lock can be acquired again.

That observation is more useful than checking that a message said “handoff complete.” The kernel's response to a separate acquisition attempt tells me whether the ownership boundary stayed closed.

The handoff also makes descriptor lifecycle part of correctness. A rejection path needs to close descriptors it will not use. A successful transfer needs to keep the chosen descriptor open long enough for the next owner. Keeping an unnecessary copy can leave a lane occupied after the process that appears to own it has finished.

The client claim helper closes extra received descriptors and returns the selected one. On refusal it closes all received descriptors. Those are small cleanup decisions with an operational consequence: leaked references can keep a real lock alive while the queue's visible state says something else.

Validate the queue handoff before keeping it

The enqueue method requires exactly one descriptor. It checks that the descriptor refers to the expected lane lock path and that the host is in dispatcher mode. It also requires a queue entry with a start and priority before retaining the new row.

My exercise covers zero descriptors, two descriptors, a descriptor for another lane file, waiter mode and entries missing the required queue identity. The rejection checks examine both the response and descriptor closure. A refusal that leaves a reference open would still change the next owner's experience.

For an accepted enqueue, I check the durable row and the status entry together. The start and priority stay unchanged. Stale slot and preclaimed-start metadata are removed. The new entry identifies the current dispatcher, and the transferred descriptor is not inheritable by default.

The inode comparison is a local consistency check within this cooperating-process protocol. Its implementation compares inode numbers, not device IDs, and it does not prove that a hostile sender actually acquired a lock before opening the file. I would keep stronger peer and filesystem validation as explicit hardening work wherever this protocol crosses a less trusted boundary.

That limitation changes how I describe the component. It is a local ownership handoff for cooperating launcher and driver processes. It is not a replacement for operating-system isolation, a capability service or a cross-host distributed lock.

Admission creates a claimable attempt

Queued and starting are different states. A queued row is still waiting for admission. A starting row is being offered to a driver, with a token that identifies the intended claim.

The claim guard in the retained implementation is direct:

            if not row or row["state"] != "starting" or row.get("token") != message.get("token"):
                return {"ok": False, "error": "no admitted row for this token"}, ()

A matching lane name is therefore insufficient. The row must be in the right state and the claim must carry the expected token.

I test a successful claim, a wrong token, a queued row and an absent row. The wrong-token case must leave the descriptor and durable row with the dispatcher. A successful claim removes the row from the dispatcher's in-memory and SQLite queue and returns the still-open descriptor for transfer.

This is where the responsibilities change. Before claim, the dispatcher owns the queue record and the lock reference. After successful transfer, the driver owns the received reference and the work it is about to supervise. Neither a status string nor an agent's confidence substitutes for that transition.

The token is an attempt-specific check inside this local protocol. It does not grant business permissions, authorize arbitrary tools or approve the final code. Those authorities have to remain in the systems that own them.

Give an unclaimed start a deadline

An admitted start can stall before the driver takes ownership. Without a bounded recovery path, the scheduler can reserve capacity for work that is no longer making progress.

The retained dispatcher uses a 60-second claim interval. The expiry predicate is strictly greater than that interval. At exactly 60 seconds the row is still starting; immediately after it, the dispatcher returns it to queued and clears the old token.

My exercise checks both sides of that boundary. It also checks that the original queue start and priority survive the transition. The recovery changes the attempt state without pretending the work just arrived.

That preservation matters for operations. If every retry resets waiting time, the oldest struggling work can repeatedly lose its place. If an expired token remains valid, an old driver may arrive after the queue has moved on and claim an attempt it no longer owns.

I do not infer global fairness from preserving a row's start time. Fairness still depends on the scheduler's ordering, reserved capacity, workload classes and the treatment of long-running jobs. The local contract is narrower and testable: an unclaimed start returns to the queue after the configured interval, carrying its original position metadata and no reusable old claim token.

Recover intent without stealing ownership

A persistent queue row and a kernel lock answer different questions after a restart.

The row records what the dispatcher intended to hold. The lock probe establishes whether another local owner currently holds the lane. Recovery reads the stored rows and tries to take each lane's lock nonblockingly.

If the lock is available, recovery can retain the row again. A stored starting state becomes queued and loses its old token. The entry is rewritten with the recovering dispatcher's identity and current state time.

If another owner holds the lock, recovery drops the stale queue row rather than treating its database record as permission to steal the lane. That existing owner may be a driver that successfully took over before the dispatcher stopped.

I test both outcomes with real locks and a real SQLite store. In one case the old descriptor is closed and recovery reacquires the lane. In the other, a descriptor remains held outside the recovered row map; recovery leaves it alone and removes the stale persistent queue record.

This is an especially useful pattern for systems built around agents. Persistent intent should help recovery find unfinished work, but it must not override current ownership simply because a stored row looks authoritative. The right decision comes from combining the retained intent with an observation of the resource that actually mediates ownership.

Keep the queue record and status view distinct

The implementation stores queue rows in SQLite and rewrites a per-lane JSON entry for status and heartbeat consumers. Those two representations have different jobs.

The row contains the queue's retained state. The entry helps existing tools observe liveness and placement. The dispatcher rewrites that entry with its own PID and heartbeat so consumers can see who is maintaining the wait.

I would ask each adjacent consumer which representation it uses and what conclusion it is allowed to draw. A heartbeat can show that bookkeeping is running without proving useful execution. A row in starting can show an admission attempt without proving the driver received its descriptor. A worker PID can show a process without proving the requested task completed.

That is why I want state names that preserve the intermediate obligations. Combining them all into running makes dashboards simpler while making recovery decisions harder.

For operational metrics, I would keep queue wait, admission-to-claim time, productive execution, review wait and human acceptance separate. The exercise here does not measure those durations across production tasks. It establishes the state boundaries that would let those measurements mean something when collected.

The queue is also a data boundary

A long-lived dispatcher should not persist a launcher's entire environment as a convenience.

The retained helper uses an allowed-prefix expression, a rejection expression for secret-shaped variable names and a value-length bound. The exercise keeps ordinary lane and author settings, rejects unrelated variables and secret-shaped names, accepts a 4,095-character value and rejects a 4,096-character value.

This is a deliberately scoped mechanism. It filters variable names and length. A credential hidden inside an innocently named value would require a different control. I describe it as environment minimization and name-based filtering, rather than promising that it detects every secret.

That gives reviewers something concrete to reason about. Which settings does the driver actually need? Which must be supplied through a separate credential path? Which values belong in durable queue storage at all? A new model or harness should not silently widen that interface by inheriting whatever happened to be present in its parent process.

The same approach applies to task metadata. Store the fields needed for ownership and recovery. Keep private prompts, customer material and credentials outside operational records unless a defined feature and authority require them.

A successful handoff still has an uncertainty window

A transport reply can be lost after the sender has changed its state. The dispatcher may remove the row while preparing to give the descriptor, and the driver may not observe a usable successful reply. Descriptor cleanup and a later lock probe help expose the resulting state, but they do not supply an end-to-end exactly-once guarantee.

I keep that question attached to the design instead of allowing a passing local handoff test to absorb it. The attempt needs a stable identity, a recovery owner and an observation that distinguishes “driver owns it” from “no one owns it now.” Retrying should reconcile that identity rather than create a fresh task on the assumption that silence means nothing happened.

Similarly, transferring a local file descriptor says nothing about ownership on another host. Cross-host coordination needs its own authority and fencing at the protected effect. A local admission dispatcher can participate in that architecture without pretending to be the fleet's single source of every permission.

Those boundaries also help keep the supervisor useful. It can ask which owner or receipt is missing instead of launching another agent to repeat the same uncertain action.

Review the ownership map before adding capacity

The diagram I want an engineering leader to draw has distinct boxes for resource admission, engine quota, task ownership, acceptance and release. Each arrow should say what identity crosses it and what evidence closes the handoff.

For this local queue, that includes the original queue position, the held descriptor, the admitted-start token, the claim deadline and the recovery observation. For the rest of the fleet, it will include different contracts. The important thing is to keep them explicit enough that the next operator can recover work without guessing.

Choose one task and follow it from waiting to claimed ownership. Then stop the path at each boundary in a controlled fixture. Can the system identify the current owner? Can an old claim still succeed? Does the task keep its place when a start expires? Does a restart respect a lock that someone else holds?

Answer those questions before treating additional workers as additional delivery capacity. The useful improvement is a fleet whose work can move between owners without losing its identity, its limits or its path back to a decision.

How this was verified

The packet pins harness commit ff4b9a70bcfb1e8395a90b7305fedd5a68b186d1 and extracts 19 complete definitions and constants verbatim from admission.py. The exercise injects external callbacks and runs 39 local cases with actual file descriptors, flock, Unix socket-pair descriptor transfer and SQLite in owned scratch. Three author-designed mutations remove claim-token validation, change the expiry boundary or remove secret-name filtering; all make the exercise fail, with restored control passing.

Four original in-process owning tests pass against pinned source and fixtures. The other 24 process and scheduler tests were deselected and are not claimed here. The exercise does not launch workers, call a model or provider, operate systemd, change a live registry, qualify hostile protocol input, prove cross-host fencing or establish deployed fleet behavior. Historical overhead figures in source comments remain historical operator reports, not measurements from this exercise.

The first independent review detected 23 of 33 non-equivalent planted faults; one additional mutation is equivalent because the allowed environment keys and secret-name alternatives are already uppercase. After reading the gaps, I expanded the exercise from 27 to 39 cases. An author-run same-set follow-up detects all 33 non-equivalent faults, with control and restored control passing. The original independent receipt remains attached; this follow-up is not a new independent score. Peer confirmation is retained with exact source and exercise identities. A five-second per-case assertion bounds incorrectly blocking local calls; the original reviewer classified that fault through its 180-second process timeout.

Peer replay confirmed the same-set follow-up at 33/33 non-equivalent faults detected, with the expected equivalent surviving and control exit 0. The exact source and exercise hashes match the retained artifacts. The original independent 23/33 result remains the initial score.