Vector Search hard
Your recommendation embeddings come from a two-tower model trained with an unnormalized dot-product loss, and item vector norms correlate with item popularity. The ANN index was created with cosine similarity because it was the default. Offline eval looks great, but production rankings favor obscure items. What is the correct fix?
- A Leave the index alone; cosine and dot product produce the same ranking, so the gap must be serving-side feature skew.
- B Rebuild the index with inner product (MIPS) so scoring matches training; cosine discards the norm signal the model learned.
- C Switch the index to L2 distance, which on unnormalized vectors yields the same ordering as the dot product used in training.
- D L2-normalize all vectors at ingest and keep cosine, so scores land in a bounded range and line up with the training objective.
Answer: B. Rebuild the index with inner product (MIPS) so scoring matches training; cosine discards the norm signal the model learned.
Cosine and dot product rank identically only for **normalized vectors**. Here the norm carries popularity, and cosine throws it away, so the index ranks by a different objective than the one the model learned. **Inner-product search** is harder for some ANN structures because it is not a true metric, but the index metric still has to match the training metric. L2 (C) fails because it adds a ‖x‖² term that penalizes large-norm items, the opposite of dot product. Cosine is the right choice only when the model was trained on normalized embeddings.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?