Evals And Observability hard
In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- A Rerun the end-to-end eval with the old embedding model and the new chunking, then the reverse, to see which change caused the drop.
- B Swap in a larger generator model, since a stronger model can reason over weaker context and will restore the end-to-end score.
- C Score retrieval separately, for example recall@k against labeled relevant chunks, to find out whether retrieval or generation regressed.
- D Add a faithfulness judge that checks each answer against its retrieved context, since faithfulness drops explain most RAG regressions.
Answer: C. Score retrieval separately, for example recall@k against labeled relevant chunks, to find out whether retrieval or generation regressed.
End-to-end scores mix retrieval failures with generation failures. **Separate retrieval metrics** such as recall@k and context precision show where the problem is, at the cost of labeling relevant chunks for each case. Here the release changed only retrieval components, so a bigger generator (B) pays more per request to compensate for missing context it can never recover. That approach only helps when retrieval is already good and the generator is the weak link. Ablation runs (A) also work but cost two full eval passes and still mix the two failure types. A faithfulness judge (D) measures only the generation side, so it stays blind to a recall drop.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?
- Your RAG assistant's answer-quality score dropped from 82 to 70 after a change. You only measure final answer quality. Why is that a problem for debugging?