Evals And Observability hard
Your pairwise judge always shows the candidate response first. Over 200 pairs, the candidate wins 64%. In a control run with the order reversed, so the incumbent is shown first, the incumbent wins 61%. How should you interpret and fix this?
- A The candidate is genuinely better, because a 64% win rate across 200 pairs is statistically significant whatever the presentation order.
- B The judge has position bias. Grade each pair in both orders, keep consistent verdicts, and count order-dependent flips as ties.
- C Switch to absolute 1 to 10 scoring for each response alone, since independent scoring is immune to ordering effects and more reliable.
- D Use the candidate model as its own judge, since it understands its own outputs best and resolves close comparisons more accurately.
Answer: B. The judge has position bias. Grade each pair in both orders, keep consistent verdicts, and count order-dependent flips as ties.
Whichever response appears first wins about 60%+ of the time, so the verdict reflects order, not quality. **Evaluating both orders** and treating flips as ties removes position bias, at the cost of doubling judge calls. Switching to absolute scoring (C) avoids order effects but loses the discrimination pairwise grading gives, so it fits only when you need a standalone score you can track over time. Self-judging (D) adds self-preference bias on top of position bias.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?