Evals And Observability hard
A new prompt passed the offline gate with a +3 point gain. It is now in an A/B test on 5% of traffic, which was sized to detect a 1% change in task completion within about a week. After one day, task completion is down 1.5% and the change is not statistically significant. The PM wants to ship based on the offline result. What should you do?
- A Keep the test running until it reaches the precomputed sample size, since online completion is the ground truth but takes time to read.
- B Ship it now; the offline gate passed, and a non-significant online drop after one day should be treated as noise by definition.
- C Roll back immediately, because any negative online delta, significant or not, overrides a positive result from the offline eval.
- D Score live traffic with the offline LLM judge instead, since judge scores converge within hours and correlate with completion anyway.
Answer: A. Keep the test running until it reaches the precomputed sample size, since online completion is the ground truth but takes time to read.
Offline evals are a fast proxy, and online metrics are the slow ground truth. Stopping an A/B test early in either direction (B or C) inflates false-positive rates through **peeking**. Waiting for the planned sample size delays the launch but gives a valid read. Shipping on offline results alone is reasonable only when the online metric cannot reach significance in a useful time, such as for very low-traffic features.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?