Evals And Observability medium
Your team is starting work on an LLM feature that summarizes customer-support tickets. No prompts are written yet. The PM suggests waiting until launch and collecting 10,000 production logs before building any evals. What is the better decision?
- A Wait for production traffic, because eval cases written before launch cannot reflect the real input distribution and will mislead prompt iteration.
- B Adopt a public summarization benchmark with over 1,000 items now, since a large standardized set has lower variance than any hand-built set.
- C Hand-curate 50 to 200 representative and edge cases now, gate every prompt change on them, and grow the set from production failures.
- D Skip a fixed set and have engineers eyeball a few fresh outputs after each change, since early prompts churn too fast to justify curated cases.
Answer: C. Hand-curate 50 to 200 representative and edge cases now, gate every prompt change on them, and grow the set from production failures.
Building a **50 to 200 case eval set before the system exists** gives every prompt change a gate from day one. The tradeoff is that early cases may not match real traffic, and you fix that by adding production failures over time. Waiting for production logs (A) is tempting, but it means weeks of unmeasured iteration. A public benchmark (B) is the right tool for comparing general model capability, not for gating a specific product.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?