Vector Search medium
After tuning HNSW, a team declares success because the top-10 results 'look relevant' in spot checks. You need to know whether the index actually meets a 95% recall@10 target. How should you measure it?
- A Run a sample of real queries through exact flat search to get ground truth, then compute overlap with the ANN top-10.
- B Compare ANN results against human relevance labels, since recall@10 measures how many relevant documents users see.
- C Use the similarity scores directly, since an average top-10 score above 0.9 shows the index found the true neighbors.
- D Rely on the library's published benchmark recall for the same M and efSearch, since recall depends only on those parameters.
Answer: A. Run a sample of real queries through exact flat search to get ground truth, then compute overlap with the ANN top-10.
**ANN recall** measures how well the index approximates exact search, so the ground truth must come from a **flat scan** over your own data and queries. This isolates index error from embedding quality, at the cost of an offline brute-force run. Human labels are tempting, but they measure end-to-end relevance and mix model quality with index error. Labels are the right tool when you are evaluating the embedding model, not the index.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?