Vector Search medium
You need to serve 1 billion 768-dimension fp32 vectors. A full in-memory HNSW index would need several terabytes of RAM. The p99 target is 50 ms, and recall@10 must stay above 95%. What is the most cost-effective design?
- A In-memory HNSW across many large-RAM nodes, since any disk access will push query latency far past the 50 ms p99 budget.
- B Flat brute-force search over memory-mapped files on SSD, since sequential reads are fast enough to scan a billion vectors.
- C A disk-based graph index on NVMe, with compressed vectors in RAM for navigation and full vectors on SSD for reranking.
- D IVF-PQ held entirely in RAM with no reranking, since PQ codes alone reliably reach 95%+ recall at billion-vector scale.
Answer: C. A disk-based graph index on NVMe, with compressed vectors in RAM for navigation and full vectors on SSD for reranking.
A **disk-based graph index** keeps small compressed vectors in RAM to guide traversal and reads full vectors from NVMe for a handful of reranks. This trades a few milliseconds of extra latency for an order-of-magnitude cut in RAM. In-memory HNSW is tempting, but it needs about 3 TB of raw vectors plus 1.5x to 2x graph overhead, while a few SSD reads fit easily within 50 ms. In-memory HNSW wins when the SLA is in single-digit milliseconds.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?