Rag And Retrieval hard
You are switching to a new embedding model with better recall. The index holds 300M chunks, and re-embedding takes about a week. A teammate proposes upserting new-model vectors into the existing index as chunks are re-embedded, so quality improves gradually. Why is this wrong, and what should you do instead?
- A Vectors from different models are not comparable. Build a parallel index, backfill it, shadow-evaluate recall, then cut over.
- B Nothing is wrong if both models share the same dimension, because cosine similarity normalizes away the differences between the two embedding spaces.
- C Mixing only fails for HNSW. Switching the index to IVF-flat during the transition makes old and new vectors safely comparable to each other.
- D Nothing is wrong as long as each query is embedded with both models and the higher of the two scores is kept for every candidate in the index.
Answer: A. Vectors from different models are not comparable. Build a parallel index, backfill it, shadow-evaluate recall, then cut over.
Each model defines its own vector space. Matching dimensions does not make old and new vectors comparable, and scores from two models are not calibrated against each other. Incremental upserts work for content changes under a fixed model. A model change is the counter-case where a **full re-embed into a parallel index** is required, at the cost of roughly double storage during the migration.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?