Evals And Observability medium
Your team wants to replace human grading of a 200-case eval set with an LLM judge so the eval can run as a CI gate on every prompt change. What must you do before trusting the judge's scores?
- A Run the judge three times per case and average the results, since high self-consistency across samples is sufficient evidence of judge accuracy.
- B Use the strongest available model as the judge, since frontier-level reasoning transfers directly to agreement with human graders on any rubric.
- C Require a chain-of-thought rationale before each score, since visible reasoning makes the judge's verdicts auditable and therefore trustworthy.
- D Write an explicit rubric, collect human labels on a calibration subset, and confirm roughly 80 to 90% agreement before relying on the scores.
Answer: D. Write an explicit rubric, collect human labels on a calibration subset, and confirm roughly 80 to 90% agreement before relying on the scores.
An LLM judge is only as trustworthy as its measured **agreement with human labels** on a calibration set, scored against a clear rubric. The tradeoff is the up-front cost of human labeling in exchange for cheap, repeatable grading afterward. Self-consistency (A) shows the judge is stable, not that it is right, because a judge can be consistently wrong. A stronger model (B) often helps, but it still must be calibrated, since capability does not guarantee it applies your rubric the way your users would.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?