Rag And Retrieval hard
You run BM25 and dense retrieval in parallel and merge the results by adding each raw BM25 score to the cosine similarity. Lexical hits dominate every query, even purely conceptual ones. What is the most robust fusion method?
- A Reciprocal Rank Fusion, which combines results by rank position rather than raw score, so the two scorers' different scales stop mattering.
- B Multiply cosine similarity by 10 to bring it into BM25's typical range, then add the two scores as before to produce the final ranking.
- C Route conceptual queries, detected by a classifier, to dense-only retrieval, and keep raw score addition for all the remaining queries.
- D Normalize BM25 by dividing by the maximum possible BM25 score across the corpus, then sum with cosine similarity for the final ranking.
Answer: A. Reciprocal Rank Fusion, which combines results by rank position rather than raw score, so the two scorers' different scales stop mattering.
BM25 scores are unbounded and their range shifts per query with length and term rarity, while cosine similarity is bounded. Any fixed scale factor is therefore wrong for most queries. **RRF** is robust because it uses only ranks, at the cost of discarding score-magnitude information. If you have labeled data, per-query normalized weighted fusion tuned on that data can beat RRF.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?