Filtering After Retrieval Leaks Your Documents. Put the ACL Inside the Query.

Most RAG pipelines I have read do access control in the wrong place. The shape is almost always the same: embed the question, retrieve the top-k nearest chunks, then filter out the ones the current user is not allowed to see, then hand what survives to the model. It looks reasonable. It passes a demo. And it leaks.

I want to explain why the order of those two steps, retrieve then filter, is the bug, and why the access-control predicate has to live inside the retrieval query rather than after it. I built a small permission-aware RAG system called vaultrag partly to put a number on this, and the number is stark: with the access-control predicate inside the query the leak rate is 0%, and if you remove that one predicate the leak rate jumps to 81.8% while every other quality metric stays perfect. That gap is the whole point of this post.

What "filter after retrieval" actually does

Say alice is in engineering. She asks about the quarterly bonus policy. The retriever embeds her question and pulls the top-k chunks by similarity from the entire corpus, including HR's salary bands and the CEO's private compensation notes, because those are extremely relevant to a bonus question. Only then does a filter step drop the chunks alice cannot see.

The trouble is that "drop it at the end" assumes the only thing that matters is what reaches the final answer. It is not. By the time you filter, the unauthorized chunk has already been fetched into your process, and everything upstream of the filter has already touched it.

Failure mode one: components touch data they should not

Between retrieval and the filter, real systems do a lot of work, and each stage is a place the data can escape.

• The embedding and similarity math already ran over it. The unauthorized chunk was scored. That score is a signal derived from privileged content.
• The ranker or reranker sees it. If you use a cross-encoder reranker, the model literally reads the forbidden chunk's text to score it against the query. That is the protected content passing through a model that had no business reading it.
• The observability layer captures it. This is the one that bites in production. Retrieved chunks land in log lines, trace spans, error reports, and debugging dumps. A single exception thrown after retrieval but before filtering can serialize the whole candidate set into your logging backend, where it now sits with different, weaker access controls than the source document had.
• Prompt assembly can grab it a step too early. In more than one codebase I have seen the context string get built from the raw retrieved set and the filter applied to a parallel list, so the assembled prompt quietly includes text the response filter thought it had removed.

None of this shows up in the answer. The answer looks clean. The leak happened in the plumbing.

Failure mode two: top-k eviction hides authorized documents

Now the subtler failure, and the one that is pure quality loss rather than a security hole. Top-k is a fixed budget. If k is 5, you get five candidates, and if three of them are documents this user cannot see, the filter deletes three, and the user is answered from two.

The document that actually should have answered the question might have ranked sixth, just outside the window, evicted by three chunks the user was never allowed to see in the first place. The unauthorized documents did not just fail to help; they actively pushed the authorized answer out of the results. The user gets a worse answer, or a refusal, and there is no signal anywhere that this happened. The retriever did its job, the filter did its job, and the outcome is silently degraded.

The fix for this is not "increase k."

Failure mode three: over-fetch

The instinct, once you notice eviction, is to widen the net. Retrieve top-50 so that after filtering you still have enough. This is over-fetch, and it makes the first failure mode worse, not better. You are now pulling far more privileged content into your process, scoring more of it, ranking more of it, and logging more of it, all to compensate for a filter that should not have been at the end of the pipeline. Over-fetch trades a quality bug for a bigger security surface. It is the wrong lever.

The fix: the predicate goes inside the query

Every one of these failures comes from the same root: unauthorized rows entered the pipeline at all. So do not let them in. Put the access-control predicate in the same query as the vector search and the keyword search, so the database never returns a row the user cannot see.

In vaultrag the retrieval is a single SQL query where the ACL check is a join and a WHERE clause inside the same CTE that does the vector similarity and the keyword match. A chunk the user cannot see is never selected, so it is never scored, never ranked, never assembled into a prompt, and never written to a log. It cannot leak, because it was never fetched. And top-k is now computed over only the authorized set, so eviction by forbidden documents cannot happen and recall holds up without over-fetching.

A few things that make this hold together in practice:

• Deny by default. A document with no ACL rows is visible to nobody. Fail-open here means one ingestion bug publishes a document to the whole company.
• Groups come from the database, not the request. The caller says who it is; the server looks up what that means. If a client could assert its own group membership, the ACL would be decorative.
• Access is evaluated per query, not baked into the index. Revocation takes effect on the next question instead of waiting on a reindex.

The number

I did not want this to be a slogan, so vaultrag has an eval that runs a gold set of (user, question, what they should and should not see) against a real Postgres corpus and reports two metrics that only mean anything together: leak rate and recall. With the predicate in the query: 0% leak, 100% recall. Delete the predicate and re-run: leak rate goes to 81.8%, and recall does not move.

Read that second half again. The broken build answers every question completely and correctly, while handing alice the CEO's notes and HR's salary bands. A quality-only eval scores that build perfect. That is exactly why leak rate is worthless reported alone, and why I pair it with recall: retrieve nothing and you fake a perfect leak rate, retrieve everything and you fake perfect recall. You have to show both.

The honest caveat

In-query ACLs are the right default, but they are not free, and they do not cover every model. When your authorization logic is more than "which principals can see this row", for example attribute-based rules that depend on request context, or field-level redaction within a document, or row-level security that has to reason across joins, expressing all of it as a single retrieval predicate gets genuinely hard, and pushing it into the vector search can defeat the index and slow queries down. The principle still holds, keep unauthorized data out of the pipeline, but the mechanism may become database row-level security policies or a filtered index rather than one hand-written WHERE clause. What is never acceptable is fetching first and sorting it out afterward.

If you want to see the query, the gold-set eval, and the before/after leak numbers, the code is at github.com/AgentPostmortem/vaultrag. Steal the eval harness even if you skip the rest; measuring the leak is what turned this from an opinion into a number.