Rag And Retrieval hard
Your RAG index over a policy wiki is rebuilt nightly by re-embedding all 5M chunks. Legal reports that a retracted policy page was still being cited 20 hours after deletion, and the cost of the nightly job keeps growing. Which design best fixes both problems?
- A Run the full re-embedding job hourly instead of nightly. That shrinks the staleness window to an hour and keeps the pipeline simple and stateless.
- B Put a TTL of a few hours on every vector, so stale chunks expire automatically and are re-embedded the next time a query touches them.
- C Consume change events from the wiki and upsert or delete chunks by document ID and version, re-embedding only changed pages.
- D Keep the nightly rebuild but add a post-generation check that removes any citation whose source page returns a 404 at response time.
Answer: C. Consume change events from the wiki and upsert or delete chunks by document ID and version, re-embedding only changed pages.
Freshness needs an **invalidation path from the source of truth** to the index. **Change events with versioned upserts and deletes** cut both the staleness window and the compute cost, at the price of building and operating a change-data pipeline. An hourly full rebuild multiplies cost by 24 and still leaves an hour of staleness. Full rebuilds remain the right tool when the chunking strategy or the embedding model changes.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You must index 100M chunks using a 3,072-dimension embedding in fp32, served from RAM via HNSW. The raw vectors alone take about 1.2 TB before graph overhead. You need to cut memory several-fold while losing at most a point or two of recall@10. Which approach fits?
- A multi-tenant SaaS app stores all tenants in one vector index with a tenant_id field. Queries fetch the top 20 chunks by similarity and then drop chunks from other tenants. Small tenants often get 0-2 results. Why does this happen, and what is the fix?
- Runbooks are split into fixed 300-token chunks with no overlap. When a procedure step starts near the end of one chunk and finishes in the next, answers come back incomplete. What is the lowest-cost fix?
- Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?
- Your product requires every RAG answer to include clickable source citations that users can verify. How should those citations be produced?
- A runbook RAG system uses fixed 500-token chunks with no overlap. Users who ask for a multi-step recovery procedure get answers that stop at step 4 of 9, and retrieval logs show the top hit ends mid-procedure. What is the most targeted fix?