Evals And Observability medium
An engineer demos a new prompt that handles five tricky examples perfectly in a live session and wants to ship it today. The system runs at temperature 0.7. What should you require first?
- A Nothing more, because passing five deliberately hard examples is stronger evidence than a high average over many easy cases in a larger set.
- B Rerun the five demo examples at temperature zero; if they pass deterministically, the prompt is validated and ready to ship to users.
- C Run the full eval gate with several samples per case against the current baseline; a few hand-picked outputs say little given high variance.
- D Ship behind a feature flag and skip the offline eval, since online metrics are ground truth and will expose problems faster than any offline set.
Answer: C. Run the full eval gate with several samples per case against the current baseline; a few hand-picked outputs say little given high variance.
LLM outputs have **high variance**, and a handful of hand-picked examples cannot show whether the prompt regressed on the many cases outside the demo. Running the full gate costs minutes but replaces anecdote with a comparison against the baseline. Temperature zero (B) makes the five examples reproducible but still says nothing about the rest of the input distribution. Shipping straight behind a flag (D) is reasonable only for low-risk features where you can tolerate learning from users over days.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?