Evals And Observability medium
A new prompt scores 4 points higher on your offline eval. Two days into an A/B test, task completion is flat and the thumbs-up rate is slightly lower, but neither difference is statistically significant yet. What should you do?
- A Ship it, since offline evals are controlled and reproducible while online metrics are noisy, making the offline gain the more trustworthy signal.
- B Let the test reach adequate sample size; online metrics are ground truth but lag, and a lasting gap means the eval set misses real usage.
- C Roll back immediately, since any decline in thumbs-up within the first 48 hours proves the offline eval is invalid and should be discarded.
- D Replace the online metrics with an LLM judge scoring live traffic, since judge scores are available instantly and avoid user feedback bias.
Answer: B. Let the test reach adequate sample size; online metrics are ground truth but lag, and a lasting gap means the eval set misses real usage.
**Online metrics** such as task completion and thumbs ratings are the ground truth for product quality, but they accumulate slowly. Offline evals are fast and reproducible, but only as representative as the eval set. Shipping on the offline gain (A) is tempting, yet if the online result stays flat or negative at full sample size, the eval set is measuring the wrong thing and needs new cases. Trusting the offline eval alone is reasonable only for changes too small or too rare to power an A/B test.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?