Vector Search hard
A legal-discovery product serves 5,000 tenants with 768-dim embeddings. The median tenant has 20K documents and the largest has 800K. Contracts require that a search never silently misses a document that a full scan would rank in the top 50. The team plans one HNSW index per tenant. What should they do?
- A Keep per-tenant HNSW but set ef_search very high, which guarantees exact top-50 results once ef far exceeds k.
- B Use exact flat search per tenant; the largest tenant is about 2.4 GB of vectors, so no graph is needed.
- C Use one global HNSW index with a tenant post-filter, amortizing graph memory and build time across tenants.
- D Use a PQ-compressed flat scan to cut memory 16x while keeping results exact, since every vector is compared.
Answer: B. Use exact flat search per tenant; the largest tenant is about 2.4 GB of vectors, so no graph is needed.
ANN indexes trade recall for speed, and no ef_search setting turns HNSW into a guarantee. At these sizes, **flat search** is both exact and cheap: the median tenant is about 60 MB, and even the largest scans in tens to low hundreds of milliseconds with parallelism, with no graph build or memory overhead. PQ (D) still compares every vector, but against lossy codes, so the ranking is approximate. HNSW would become necessary only if tenants grew toward tens of millions of vectors under tight latency budgets.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?