Evals And Observability hard
You compare two prompt variants with an LLM judge that gives absolute 1 to 10 scores. Variant A averages 7.8 and variant B averages 7.9, and 90% of all scores fall at 7 or 8. The product team needs to decide which variant to ship. What should you change in the evaluation?
- A Switch to pairwise comparison with randomized order, because relative judgments separate close variants better than compressed absolute scores.
- B Widen the absolute scale to 1 to 100 so the judge has enough resolution to separate the two variants' outputs from each other.
- C Ship variant B now, since any positive mean delta on the same fixed golden set counts as evidence of a real quality improvement.
- D Require chain-of-thought reasoning before each absolute score, which removes the clustering and makes the 0.1 difference meaningful.
Answer: A. Switch to pairwise comparison with randomized order, because relative judgments separate close variants better than compressed absolute scores.
Absolute scores from LLM judges compress into a narrow band, so small differences fall inside the noise. **Pairwise comparison** asks a simpler relative question and agrees with humans more often, at the cost of O(n²) comparisons when there are many variants. Widening the scale (B) adds false precision: the judge just clusters at 75 to 80 instead. Absolute scoring is still the right tool when you need a stable score you can track over time against a fixed rubric.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?