Evals And Observability medium
Your weekly offline eval has held steady at 86/100 for three months, with no prompt or model changes. Over the same period, the thumbs-down rate has crept from 4% to 7%. What is the most likely cause, and what should you do?
- A The stable eval shows quality is unchanged, so the rise is user noise; hold off on any action until the offline eval score itself starts to move.
- B Users are reacting to repetitive answers that a frozen eval cannot see, so raise the generation temperature to increase output diversity.
- C The weekly cadence is too coarse to catch the decline, so rerun the same golden set daily to surface the degradation much sooner.
- D Input drift: compare the live query distribution against the eval set's distribution and add sampled recent failures to the golden set.
Answer: D. Input drift: compare the live query distribution against the eval set's distribution and add sampled recent failures to the golden set.
A frozen golden set keeps measuring the same old inputs, so it cannot see **input drift** when users start asking new kinds of questions. Monitoring input distributions and refreshing the set gives up some eval stability in exchange for coverage. Treating the stable score as proof of quality (A) is the tempting error, because the eval and the live traffic have diverged. Running the same set more often (C) only helps when the model itself is changing, not the inputs.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?