Vector Search hard
An IVF index with 16K lists had its coarse quantizer trained on a 1M-vector sample 18 months ago. Since then the corpus has grown 5x, mostly from a new content type (source code) that looks nothing like the original prose. At the same nprobe, recall@10 fell from 96% to 85% and p99 latency doubled. What is the root cause, and what is the fix?
- A nprobe is simply too low for a larger corpus; raising it restores recall without touching the quantizer or the lists.
- B IVF recall inherently decays with corpus size; migrate to HNSW, since graph indexes never need training or upkeep.
- C Centroids no longer fit the data, so new vectors pile into a few huge lists; retrain the quantizer on a fresh sample and rebuild.
- D The PQ sub-quantizer codebooks drifted; retraining only them restores recall while keeping existing list assignments.
Answer: C. Centroids no longer fit the data, so new vectors pile into a few huge lists; retrain the quantizer on a fresh sample and rebuild.
IVF's **coarse quantizer** is trained on a sample. After a distribution shift, the new content lands in a handful of overloaded lists, so probing nprobe lists covers less of the true neighborhood and the giant lists inflate scan time. Retraining and reassigning costs a full rebuild but restores balanced lists. Raising nprobe (A) recovers some recall at even higher latency. It is the right lever only when lists are still balanced and the data distribution has not shifted.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?