Rag And Retrieval hard
Your RAG system over 2M support articles measures recall@50 of 94% but precision@5 of 31% on a labeled set. Answers often draw on off-topic passages. The context budget fits about 6 chunks. What is the highest-leverage change?
- A Raise the top-k passed to the LLM from 6 to 30, so the relevant passages already in the candidate set reach the context window.
- B Swap to an embedding model with a larger dimension, since better first-stage vectors will push the relevant passages into the top 5 positions.
- C Add a cross-encoder reranker over the top 50 to 100 candidates and pass only the reranked top 5 or 6 to the generator.
- D Increase chunk size from 400 to 1,500 tokens, so each of the 6 slots carries more surrounding context and is more likely to hold the answer.
Answer: C. Add a cross-encoder reranker over the top 50 to 100 candidates and pass only the reranked top 5 or 6 to the generator.
High recall@50 with poor precision@5 is the classic signature of a **missing second stage**: the right passages are retrieved but ranked badly. A **cross-encoder** scores each query and passage jointly, trading tens to hundreds of milliseconds for precision. Raising k to 30 floods the context with noise and tokens. A better embedding model is the right fix only when recall@50 itself is low.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?