Evals And Observability hard
Your team evaluates a candidate model against the incumbent with a pairwise LLM judge. The judge comes from the same model family as the candidate. The candidate wins 70% of 200 pairs, but a blind human review of 60 of those pairs shows a 52% candidate win rate. What is the most likely cause, and what is the right fix?
- A The eval set is too small; expanding it from 200 to 2,000 pairs will make the 70% judge win rate converge toward the human figure.
- B Self-preference bias: the judge favors its own family's outputs. Use a judge from a different family and recheck agreement with human labels.
- C Position bias alone explains the gap; randomizing answer order will close it without any change to which model family serves as the judge.
- D Human raters are noisier than the judge at scale; trust the 70% figure because the judge applies the rubric more consistently across all pairs.
Answer: B. Self-preference bias: the judge favors its own family's outputs. Use a judge from a different family and recheck agreement with human labels.
LLM judges show **self-preference**: they rate outputs that match their own style higher, so a judge from the candidate's family inflates its win rate. A cross-family judge calibrated against a human-labeled set costs an extra calibration step but removes the conflict of interest. Randomizing order (C) fixes position bias, but that bias affects both models symmetrically and would not produce an 18-point gap that only favors the judge's own family.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?