Rag And Retrieval hard
A team retrieves the top 40 chunks and passes all of them to the model because 'the context window can hold it.' Compared with top-8, quality dropped and latency rose, and answers often miss facts from chunks placed mid-context. What assembly strategy is best?
- A Keep all 40 but sort them by document date, so the model sees a coherent chronological narrative and can locate facts anywhere in the context.
- B Rerank to a small set, drop near-duplicates, and put the strongest evidence at the start or end of a budgeted context.
- C Keep all 40 and first summarize each one with a smaller model, so the same breadth of evidence fits in fewer tokens without losing any facts.
- D Raise k to 80 and switch to a model with a larger context window, since the quality drop means relevant evidence was still being left out.
Answer: B. Rerank to a small set, drop near-duplicates, and put the strongest evidence at the start or end of a budgeted context.
A higher k adds noise, tokens and latency, and models attend least reliably to evidence buried mid-context. A tight, reranked, deduplicated context trades breadth for precision. Per-chunk summarization adds a model call per chunk and is lossy. It is the right choice for aggregation questions where the evidence is genuinely spread across many documents.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?