Vector Search hard
Your team is choosing ef_search for a production HNSW index and needs to report recall@10 for each candidate setting. Which methodology measures ANN recall correctly?
- A Treat the same index at ef_search 10x higher as ground truth, since a wider search converges on true neighbors.
- B Use click-through rate on the top 10 results as recall, since users click whatever the true nearest neighbors are.
- C Generate random query vectors uniformly over the space, run exact search for them, and compare to the index.
- D Sample real production queries, compute the exact top 10 by flat search on the same snapshot, and compare each setting.
Answer: D. Sample real production queries, compute the exact top 10 by flat search on the same snapshot, and compare each setting.
Recall measures how closely the index agrees with exact search. Ground truth must therefore come from a **brute-force scan** over the same corpus snapshot, using queries drawn from the real distribution. That costs a full scan per query, which is why it is done offline on a sample. Option A is circular, because nodes the graph cannot reach are missing at every ef, so it overstates recall. CTR (B) measures embedding relevance, which is a separate question from ANN fidelity.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?