Evals And Observability hard
Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A Online metrics simply lag offline ones; the 19-point offline gain will show up in task completion within the next quarter or so.
- B The judge is miscalibrated; replace it, since a well-calibrated judge would have shown the same flat trend that the online metrics show.
- C Grow the golden set to 1,000 cases drawn from the same original source, so the offline score becomes more statistically robust.
- D The prompt has overfit the golden set. Keep a held-out split refreshed from production traffic, and gate releases on that split.
Answer: D. The prompt has overfit the golden set. Keep a held-out split refreshed from production traffic, and gate releases on that split.
Tuning repeatedly against one eval set turns it into a training set, so gains stop transferring. A **held-out split refreshed from production** keeps the gate honest, at the cost of continuous labeling. Growing the set from the same source (C) improves statistical precision but keeps the same blind spots, and 40 rounds of selection still overfit to it. Waiting for lagging metrics (A) is reasonable only when the online signal needs weeks to accumulate, not after two months.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?
- Your RAG assistant's answer-quality score dropped from 82 to 70 after a change. You only measure final answer quality. Why is that a problem for debugging?