Authorized recall: the retrieval budget has to belong to the caller

Our memory retriever has a five-hundred-candidate limit. It also runs an exact ranking pass inside the caller's scope, alongside approximate candidate generation. Those choices belong together: I want bounded work without letting another tenant's corpus consume the caller's opportunity to find evidence.

The less obvious consequence is what happens later. Source freshness is checked after ranking, and that work is bounded too. A perfectly isolated result can still be empty while useful authorized evidence exists beyond the candidate boundary.

I built the retrieval path around explicit scope, honest fallback modes and source verification. The operator problem is to preserve those properties while deciding how much retrieval work and context a request can spend.

This came up again in the October 17 planning material. The proposed pipeline combined fifty dense candidates and fifty lexical candidates, fused ranks, reranked fifty, kept six to eight passages and capped answer context at roughly six thousand tokens. Our existing memory service uses a different five-hundred-candidate implementation. I treat those as two designs to reconcile, not measurements from one deployed system.

The useful question is what each budget means and which evidence can disappear at each boundary.

Give recall an authorized denominator

A retrieval score on an unrestricted corpus can answer the wrong question for a multi-tenant assistant. The documents most similar to a query may belong to another organization, a private conversation or a source the actor can no longer access.

I define the useful denominator as known relevant evidence the current request may use. That requires an actor, scope, time and source state alongside the query. “Relevant document” is incomplete without those inputs.

The simplest list model makes the difference visible. I constructed five authorized relevant documents and five hundred foreign documents with higher declared scores. Taking the global top five hundred and then filtering by scope returned zero authorized documents. Filtering by scope before ranking returned all five. Both returned zero unauthorized documents.

That is a synthetic ordering exercise, not an HNSW benchmark. It demonstrates a distinction that a leakage-only metric misses: the first branch is safe from this disclosure while failing to retrieve the caller's evidence.

I also changed the fixture so every document belonged to the caller. The late-filter branch then returned five hundred candidates. That control matters. It establishes that the empty result depends on the scope-and-cap interaction rather than a model that always returns nothing.

Scope must reach the candidate query

Our memory search resolves the principal and context, constructs visibility conditions, and carries those conditions into lexical and vector candidate generation. Visibility includes organization and relevant project, person or conversation scope. It also accounts for the source's project and conversation and the memory's approval state.

The service runs memory-table work through a restricted database role. It checks that the role cannot log in, bypass row security or act as a superuser. Missing safe configuration produces an unavailable-storage error instead of quietly using the application's broader connection authority.

This is a useful defense boundary, but I keep application scope and database enforcement visible as separate mechanisms. A SQL predicate, a restricted role and a current source check contribute different evidence. A deployment needs to establish that its policies, role and queries agree.

For candidate recall, the central relationship is compact:

const approximate = await db.select({ id: memoryItems.id })
  .from(memoryItems).where(predicate)
  .orderBy(distance).limit(MEMORY_CANDIDATE_LIMIT);

const exact = await db.select({ id: memoryItems.id })
  .from(memoryItems).where(predicate)
  .orderBy(sql`(${distance}) + 0`)
  .limit(MEMORY_CANDIDATE_LIMIT);

for (const row of [...approximate, ...exact]) ids.add(row.id);

That excerpt is adapted from our implementation. The exact expression is deliberately shaped to avoid using the approximate index for that pass. The two result sets are combined by identifier. It is a scoped fallback for candidate generation, not a promise to retrieve every relevant fresh fact in the database.

The pgvector documentation on filtering and multi-tenancy explains why approximate-index filtering and shared populations can affect recall. Our specific query plans still need inspection in the actual database and extension version.

Exact has a scope and a limit

“Exact” is easy to overread. It describes ranking under a particular distance expression and predicate. It does not establish that our embedding represents every relevant concept, that five hundred candidates cover all useful evidence or that subsequent source checks preserve enough results.

Our helper sets iterative HNSW scanning to strict ordering and an ef_search value of one hundred. It then adds the separate exact scoped pass. These are implementation choices. I would compare plans and outcomes before deciding whether a different tenant size, partition scheme or workload warrants another strategy.

There is a cost to the exact pass. It can make candidate work depend on the size of the authorized corpus. The limit bounds returned rows; it does not by itself bound how many rows a database must consider while ranking. Our restricted-role work also sets a two-thousand-millisecond statement timeout. That is a configured bound, not a measured query latency or total request deadline.

I want the system to distinguish an empty valid result from an unavailable retrieval path and an exhausted work budget. Otherwise the assistant may convert infrastructure constraints into an unjustified factual answer.

Count the operations behind the constant

Five hundred is not one global work allowance. It bounds a candidate query's returned rows and the later selected-source loop. Intermediate unions can be larger.

With both vector streams enabled, the helper can run approximate and exact passes for local vectors and for compatible semantic vectors. A loose combinatorial ceiling is two streams times two passes times five hundred identifiers: two thousand before deduplication. The lexical query can contribute another five hundred entries before fusion. Actual overlap reduces that pool; these are derived upper bounds, not measured result sizes.

The service then scores the dense candidates, fuses ranks and checks at most five hundred selected sources. Those are distinct populations. A benchmark of five hundred cosine operations would not establish an end-to-end bound for all query plans and intermediate sets.

I would record each stage's input and output count, including deduplication and rejection reasons. That makes a budget regression attributable: did the database inspect more rows, did a union expand, did source verification become expensive or did context packing repeat the same evidence? One constant cannot answer those questions.

Capability belongs to the database being queried

Our vector-capability helper checks for the local and semantic HNSW indexes in the current database. It does not reuse a process-global answer from a previous database.

This matters when one service process talks to multiple databases or tests create different shapes. An index existing somewhere in the process's history does not establish that the current query can use it. Capability is also distinct from authorization: a database may have an index while an organization is not entitled to dispatch a semantic provider request.

In our search path, the presence of the local index determines whether the vector-candidate helper is used. Without it, the fallback takes a bounded recency-ordered set of visible rows and scores locally. That preserves a useful route, but it changes which older records can become candidates.

I report that mode instead of describing every successful response as semantic search. A healthy response with degraded coverage is still a product state the caller may need to understand.

Name the local fallback honestly

Our credential-free embedding is called local-feature-hash-v1. It produces one hundred twenty-eight dimensions by normalizing text, hashing tokens into signed bins and normalizing the vector. It is explicitly not a pretrained semantic model.

The offline exercise executes that actual function. Repeated input produces the same vector. A nonempty example has unit norm, its cosine with itself is one, and empty or dimension-mismatched inputs follow the helper's defined zero-result behavior.

Those properties establish determinism and arithmetic behavior. They do not establish paraphrase understanding. Token collisions and lexical overlap can change rankings in ways a neural embedding would handle differently. I would never present the local route as a cheaper model with measured equivalent semantic quality merely because both routes produce arrays of numbers.

The optional semantic index path expects seven hundred sixty-eight dimensions and a matching model identifier. The adapter's generic batch validation also checks the number of vectors, finite values and declared dimensions. A technically valid batch still has to match the particular index contract.

In the exercise, a local fake provider returns one valid batch and six malformed alternatives. The actual validation helper rejects the model mismatch, wrong vector count, zero dimensions, excessive dimensions, inconsistent vector length and nonfinite value. Seven calls go to the fake function; zero calls go to a model service.

Model identity matters during upgrades. Two models can emit the same dimension count while representing different spaces. Reuse of stored vectors requires an explicit compatibility argument or a backfill, not only a length check.

Fuse rank without laundering confidence

Our retriever uses reciprocal rank fusion to combine lexical and vector lists. The implementation adds one divided by sixty plus rank plus one for each occurrence, then applies a deterministic identifier tie-break.

for (const list of [lexical, vector]) {
  list.forEach((id, rank) => scores.set(
    id, (scores.get(id) ?? 0) + 1 / (60 + rank + 1)
  ));
}

A document present near the top of both lists can outrank one appearing in only one. The offline fixture verifies that behavior with a shared document and verifies deterministic ordering for a tie.

This is rank aggregation. The fused number is not a probability that the document is true or relevant. Our local vector threshold of 0.20 is also a selection rule, not a calibrated confidence statement. The answer layer needs source evidence and an acceptance policy beyond the numeric rank.

I preserve provenance through this step: source identity, version, scope and retrieval mode. A top-ranked text fragment without its source context is hard to revalidate and easy for a later summarizer to overstate.

Freshness can spend the remaining budget

The active-memory query already filters validity windows, expiry and verification metadata. The service then performs current source checks on ranked candidates. Those checks can reject a row whose stored metadata looked eligible but whose source has changed or become inaccessible.

The loop considers at most five hundred selected candidates and stops when it has enough valid results for the requested page. That bounds work. It also means candidate exhaustion is a real state to reason about.

I modeled five hundred stale higher-ranked candidates followed by five fresh relevant candidates. Applying the cap before the freshness predicate returned zero fresh rows. Applying an idealized freshness predicate before the cap returned five. The model does not execute our source verification or SQL; it isolates the ordering tradeoff.

Moving every freshness decision into candidate generation is not automatically feasible. Some checks depend on live source access and content hashes. Running them eagerly over a large corpus can make the cost unacceptable. Useful alternatives include batched checks, maintained freshness metadata, bounded additional retrieval rounds or an explicit insufficient-evidence outcome.

Each needs a ceiling and an owner. A recovery loop that keeps widening until something appears can quietly defeat both latency budgets and the intended authority boundary.

Keep context packing separate from candidate recall

Candidate retrieval produces options. The answer context can hold only some of them. I evaluate those as different stages because a strong candidate retriever can still lose evidence during packing.

The October 17 proposal's roughly six-thousand-token context cap is a planning budget. It is not a measured safe context size or an implemented memory-service constant. To reason about packing without inventing tokenizer results, the exercise assigns each of the five authorized documents a synthetic weight of three hundred twenty units.

Budgets of six hundred forty, nine hundred sixty and sixteen hundred units retain two, three and five documents: forty, sixty and one hundred percent of this fixture's known relevant set. The units are neither provider tokens nor bytes. The curve demonstrates a budget relationship; it does not establish answer quality.

A production packer should use the actual route's tokenizer and include instruction, tool, citation and output-reserve overhead. It should preserve enough evidence for the claim being made, rather than fill the context with individually attractive passages that all repeat one point.

I would tag passages by the questions and claims they support, retain disagreements when material and keep source references attached. Summarizing a long passage can save context while changing which facts remain verifiable. That compression deserves its own evaluation.

Treat degradation as information

Our implementation distinguishes missing provider configuration, disabled semantic capability, provider failure and pending semantic indexing. A local fallback can remain useful under those states. Reporting the mode gives the caller a chance to adjust its expectations or ask for another route.

I would carry a similarly explicit outcome contract into the answer stage: enough fresh authorized evidence, bounded search exhausted, retrieval unavailable or no supporting evidence found within the declared scope. Those are proposed answer-level distinctions, not a claim that our current response schema exposes every one.

Abstention is useful when it prevents an unsupported answer and leaves a clear next step. It becomes an outage pattern if the system hides the reason or never recovers. The owner needs to know whether to restore an index, repair provider configuration, refresh sources or expand a permitted search budget.

Semantic-provider calls are outbound data operations too. Our memory proposal path authorizes and resolves the source before requesting an external embedding, then rechecks inside the committing transaction. Search resolves context before querying its optional provider. I still want the exact provider route and data policy included in the privacy review; relevance engineering does not exempt embedding requests from information-flow decisions.

Measure the stages, then the outcome

For an evaluation set, I would store the query, actor class, scope, time, known relevant authorized sources, expected unavailable states and the final task's evidence requirements. The same query under a different audience can have a different correct answer set.

Then I would measure candidate recall, fresh-valid recall, packed-evidence coverage and answer support separately. I would also measure unauthorized exposure at each boundary, not only in the final text. A later filter cannot undo disclosure to an external reranker or provider.

The performance record should include candidate counts, source-check work, provider route, token usage, query-plan evidence and latency distribution for each corpus shape. An exact scoped pass may be cheap for one tenant and expensive for another. A fast lexical fallback may miss an older relevant passage. Those are measurable tradeoffs, not properties to infer from a mode name.

The acceptance control matters as much as the score. Add a foreign population without changing the caller's relevant set. Add stale high-ranked records without changing the fresh answer. Change model identity while preserving vector dimensions. Confirm that an ordinary authorized query still works. Each mutation isolates a failure pattern instead of rewarding one implementation's happy path.

Start with one known answer set

Take a query for which you can identify the relevant evidence before running retrieval. Fix the caller, scope and source state. Then inspect what survives candidate generation, freshness checking and context packing.

If evidence disappears, identify the budget that removed it. If the evidence is unavailable, give the caller a specific bounded result. If the system wants to widen scope, make that an authority decision rather than a relevance optimization.

That is the contract I want for an assistant's memory: retrieve evidence the request may use, preserve the reason it is usable and spend an explicit amount of work to do it. The model gets a better foundation when retrieval admits what it knows about its own limits.

How this was verified

Our existing memory retrieval and service source provide the implementation mechanisms; their relevant history predates the reviewed week, with retrieval changes on October 7 and 8. A retained October 17 session supplies the proposed fifty-plus-fifty candidate and roughly six-thousand-token design. On October 19, an offline Node exercise executed copied local embedding, cosine, fusion and provider-validation functions, plus separately declared list, freshness and packing models. It made no network, database or real model calls. The toy corpus, relevance scores and packing weights are synthetic. Current SQL plans, RLS behavior, production recall, latency, spend and deployment coverage were not measured. Hashes, code and receipts are retained in the private package.