Rag And Retrieval medium
An enterprise assistant indexes HR, finance, and engineering documents in one shared index. It retrieves the top 10 chunks, generates an answer, and then a post-processor removes citations to documents the user cannot access. A security review flags this design. Why, and what is the fix?
- A The post-filter is fine, but it should also redact sentences that quote restricted documents before returning.
- B The model already read restricted content; enforce ACLs as a filter inside the retrieval query so it never enters the prompt.
- C The post-filter is fine as long as the system prompt instructs the model never to reveal content from restricted documents.
- D Restricted chunks should be retrieved but summarized first, so the generator only sees a high-level paraphrase of them.
Answer: B. The model already read restricted content; enforce ACLs as a filter inside the retrieval query so it never enters the prompt.
Once restricted text is in the context, it can leak through paraphrase or inference, and no output filter or prompt instruction reliably catches that. **Filtering at retrieval time** requires syncing ACL metadata into the index, and it can reduce per-user recall. It is still the only safe boundary. Post-generation filtering is acceptable only when every user has identical access to the whole corpus.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?