Evals And Observability medium
Your RAG assistant's end-to-end correctness score dropped 8 points after one release. That release changed the chunking strategy, the embedding model, and the answer prompt. You track only a single end-to-end score. What should you add?
- A Score retrieval (recall@k against gold passages) separately from generation (faithfulness to retrieved context) to isolate which stage regressed.
- B Grow the end-to-end eval set from 150 to 2,000 cases so the drop can be attributed to a component with much tighter confidence intervals.
- C Have an LLM judge rate overall helpfulness on a 1 to 10 scale, since a finer-grained absolute score will reveal which component caused the drop.
- D Roll back all three changes, then reapply them in production one at a time and use the thumbs-down rate to identify which change caused the drop.
Answer: A. Score retrieval (recall@k against gold passages) separately from generation (faithfulness to retrieved context) to isolate which stage regressed.
A single end-to-end score tells you *that* quality dropped but not *where*. Separate **retrieval metrics** and **generation metrics** localize the regression, at the cost of maintaining gold passage labels. A larger end-to-end set (B) shrinks noise but still mixes the two stages together. Sequential online rollouts (D) can isolate the cause, but each change takes days and users are exposed to regressions that an offline split would catch.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?