Vector Search hard
You serve RAG over 200M chunks embedded with your current model (1,536 dims). A new embedding model, also 1,536 dims, scores better on your eval set. An engineer proposes switching the query encoder to the new model today and re-embedding chunks lazily as they are edited. What should you do instead?
- A Build a separately versioned index with the new model, dual-write during backfill, and cut over after validating recall.
- B Proceed with the lazy plan, since equal dimensionality means both models share one coordinate space and comparable scores.
- C Swap the query encoder now and rely on the HNSW graph, which is model-agnostic, to keep neighborhoods valid during re-embedding.
- D Upsert re-embedded vectors into the existing index in place, keeping one index and avoiding doubled memory during the migration.
Answer: A. Build a separately versioned index with the new model, dual-write during backfill, and cut over after validating recall.
Two embedding models produce unrelated vector spaces even at the same dimension. Any mix of old and new vectors returns noise, and so do new-model queries against old documents, which makes a model swap a **full reindex**. A **versioned index** with dual-writes costs roughly double the memory and compute during backfill, and in return gives an atomic, reversible cutover. In-place upserts (D) look cheaper but leave the index in a mixed state for the whole backfill. They are acceptable only if search can be taken offline until re-embedding finishes.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are building an IVF index on 20 million vectors. A teammate asks why the build needs a 'training' step before any vectors can be added. What is the reason?
- An HNSW index serves a marketplace where about 30% of listings are updated or removed each week. Each update is applied as a delete plus an insert, and deletes mark tombstones. Over a month, recall@10 drifts from 97% to 89% and p99 latency rises at the same ef_search. What is happening, and what should you do?
- Users delete documents in your primary database, but the deleted documents keep appearing in vector search results for hours. The vector index is rebuilt nightly. What is the best fix?
- Your embedding model outputs unnormalized vectors whose magnitudes vary widely. A teammate wants to switch the index from cosine similarity to dot product because it is cheaper to compute. What is the consequence?
- You are setting an SLO for a new ANN-backed recommendation search over 50 million vectors. A product manager asks for 100% recall@10 with single-digit millisecond latency. What is a realistic response?
- A B2B SaaS app serves 500 customers, and every query searches only one customer's documents. One large customer runs heavy batch queries that slow down search for everyone. Which sharding approach fits best?