Evals And Observability medium
Your CI eval gate scores jumped 5 points overnight, with no prompt or model change on your side. The LLM judge is called through a 'latest' model alias. What should you change?
- A Pin the judge to a fixed model version, and re-validate it against the human calibration set whenever you deliberately upgrade it.
- B Keep the latest alias so the judge benefits from improvements, and rebaseline the gate threshold whenever the scores shift.
- C Make the judge the same model as the system under test, so any provider update moves both together and the effect cancels out.
- D Average judge scores from three different providers' latest models so a version change in any one judge is diluted by the other two.
Answer: A. Pin the judge to a fixed model version, and re-validate it against the human calibration set whenever you deliberately upgrade it.
An eval gate compares scores across time, so the judge must be a **fixed model version**; otherwise a judge update looks like a change in your system. Pinning gives up automatic judge improvements in exchange for comparability, and each deliberate upgrade needs re-calibration against human labels. Rebaselining on every shift (B) is tempting, but it hides real regressions that happen to coincide with judge changes. Using the same model as the system under test (C) adds self-preference bias.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?