Vector Search hard
You must serve 2B 768-dim fp32 vectors (about 6 TB raw) with recall@10 of 95% and p99 under 20 ms. The budget allows a handful of nodes, each with 512 GB of RAM and fast NVMe SSDs. Which architecture fits?
- A In-memory HNSW across enough nodes to hold vectors and graph, since only RAM-resident indexes meet a 20 ms p99.
- B An SSD-resident graph index with PQ codes in RAM for routing and full vectors on SSD for a final exact rerank.
- C Build HNSW and memory-map it from SSD, letting the OS page cache keep hot parts in memory and fault in the rest.
- D Flat search streamed from SSD across nodes, since it is exact and sequential reads avoid random I/O penalties.
Answer: B. An SSD-resident graph index with PQ codes in RAM for routing and full vectors on SSD for a final exact rerank.
A **disk-based graph index** keeps only compressed codes in RAM and lays out the graph so each hop is one SSD read. A query then costs tens of random reads, which fits a 20 ms budget at a fraction of in-memory cost, with slower builds and slightly lower recall as the tradeoff. Memory-mapping HNSW (C) fails because its random hops become page faults with no locality, and latency collapses once the working set exceeds RAM. In-memory HNSW (A) is the right choice only when the budget covers 10+ TB of RAM.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?