Rag And Retrieval hard
Your p95 retrieval budget is 500 ms. First-stage hybrid retrieval takes 80 ms, and a cross-encoder over 200 candidates adds 450 ms. Offline, first-stage recall is 91% at N=50, 93% at N=100 and 94% at N=200. What is the best adjustment?
- A Drop the reranker and pass the first-stage top 20 to the model, since the LLM can make its own relevance judgments within the context window.
- B Keep reranking 200 candidates but move the reranker after generation, rescoring the answer's citations so the latency is hidden from users.
- C Rerank only the top 50 to 100, trading 1 to 3 points of candidate recall for a reranking cost that fits the budget.
- D Replace hybrid retrieval with dense-only retrieval to save first-stage time, freeing enough budget to keep reranking all 200 candidates.
Answer: C. Rerank only the top 50 to 100, trading 1 to 3 points of candidate recall for a reranking cost that fits the budget.
Cross-encoder cost scales roughly linearly with candidate count. Cutting from 200 to 50 to 100 brings it to about 110 to 225 ms, and the recall curve has nearly plateaued by then. Dropping the reranker gives up precision for latency and pushes ranking onto the noisy context. If recall were still climbing steeply past 100, a smaller distilled reranker would beat shrinking N.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?