All Questions
Rag And Retrieval hard

You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?

  1. A Increase chunk size from 400 to 1,600 tokens to cut the vector count by 4x, shrinking memory proportionally with no change to the index.
  2. B Quantize in-memory vectors (int8 or product quantization) and rescore the top candidates with full-precision vectors fetched from disk.
  3. C Shard the index across four times as many nodes, so each node holds a quarter of the vectors and per-node memory pressure falls by 4x.
  4. D Keep fp32 vectors on disk only and compute exact brute-force similarity for every query, removing HNSW graph memory entirely.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions