All Questions
Rag And Retrieval hard

Your offline eval shows recall@5 of 92% on a labeled set, so the gold passage is in the context for nearly every question. Yet human graders mark 35% of answers wrong, and the wrong answers often ignore or contradict the gold passage. Where should the team invest next?

  1. A Generation: tighten the prompt and context assembly, e.g., answer only from context, reorder chunks, and abstain when unsupported.
  2. B Retrieval: switch to a higher-dimensional embedding model, since 92% recall still leaves room and better vectors will raise answer accuracy too.
  3. C Retrieval: raise k from 5 to 25, so the generator sees more corroborating passages and is less likely to contradict the single gold passage.
  4. D Chunking: shrink chunks so the gold passage is less diluted, since the model's failure to use it suggests the embedding of that chunk is weak.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions