Rag And Retrieval hard
A legal-contracts RAG system uses 2,000-token chunks with top-8 retrieval, which fills about 16K tokens per query. Answers are often vague and cite the right contract but the wrong clause, and per-query cost is high. Which change best improves precision without losing the surrounding context the model needs?
- A Grow chunks to 4,000 tokens so each clause appears alongside its definitions, and drop top-k to 4 to keep total context tokens constant.
- B Index small 300 to 500 token chunks for matching, then expand each hit to its parent section only when assembling the context.
- C Keep 2,000-token chunks and raise k to 16, since more candidates increase the chance that the exact clause appears somewhere in the context.
- D Switch to sentence-level chunks of about 30 tokens with no expansion, which maximizes embedding precision for each individual clause.
Answer: B. Index small 300 to 500 token chunks for matching, then expand each hit to its parent section only when assembling the context.
A large chunk produces one vector averaged over many clauses, so it matches the contract rather than the clause. **Small-to-big retrieval** matches at a fine granularity and supplies context at a coarse one, at the cost of maintaining parent-child mappings. Bare sentence-level chunks lose the definitions and cross-references the model needs. Bigger chunks are right only when questions concern whole-document themes.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?