Rag And Retrieval hard
You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A Increase chunk size from 400 to 1,600 tokens to cut the vector count by 4x, shrinking memory proportionally with no change to the index.
- B Quantize in-memory vectors (int8 or product quantization) and rescore the top candidates with full-precision vectors fetched from disk.
- C Shard the index across four times as many nodes, so each node holds a quarter of the vectors and per-node memory pressure falls by 4x.
- D Keep fp32 vectors on disk only and compute exact brute-force similarity for every query, removing HNSW graph memory entirely.
Answer: B. Quantize in-memory vectors (int8 or product quantization) and rescore the top candidates with full-precision vectors fetched from disk.
Memory is 100M x 3,072 x 4 bytes is about 1.2 TB. **Quantization** shrinks it 4x (int8) to well over 10x (product quantization), and full-precision rescoring of a short candidate list recovers most of the lost recall at a small latency cost. Sharding spreads the memory across more nodes but does not reduce the total. 4x bigger chunks dilute relevance. Dimension truncation is an alternative only if the embedding model was trained to support it.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?
- Policy documents live in a CMS. A nightly job embeds and inserts new documents into the vector index. Users keep getting answers that cite policies which were deleted or replaced weeks ago. What is the root cause and fix?