Rag And Retrieval medium
A contract-analysis RAG system splits documents into 2,000-token chunks and puts the top 3 in the prompt. Answers usually cite the right contract but miss the specific clause asked about, and per-query token cost is high. What should you change?
- A Index whole documents as single chunks so the model always sees complete clauses and never loses context.
- B Keep 2,000-token chunks but raise top-k from 3 to 10 so more of each relevant contract is visible to the model.
- C Keep 2,000-token chunks and add a cross-encoder reranker over the top 20, so the chunk with the right clause ranks first.
- D Re-chunk to around 400 tokens along clause boundaries with modest overlap, then retrieve more but smaller chunks.
Answer: D. Re-chunk to around 400 tokens along clause boundaries with modest overlap, then retrieve more but smaller chunks.
A 2,000-token chunk averages many clauses into one embedding. That **dilutes relevance**, and it spends context budget on text the question does not need. Smaller, structure-aware chunks improve precision but can lose surrounding context, which is why you add overlap. A reranker still scores whole 2,000-token chunks, so it cannot point at one clause inside the right contract. Bigger chunks are the right call when questions need whole-document synthesis, such as summarizing an entire contract.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?