Evals And Observability hard
Your assistant has thumbs-up/down buttons. About 2% of responses get rated, and 85% of those ratings are thumbs-up. The team proposes making thumbs-up rate the primary quality KPI and building the eval set only from thumbs-down examples. What is the main flaw, and what should you do instead?
- A There is no real flaw; an 85% positive rate across millions of requests is strong enough evidence to make it the primary KPI.
- B Raters are a biased sample. Pair thumbs with implicit signals like retries and abandonment, and label a sample of unrated traffic.
- C Explicit feedback is unreliable, so drop it entirely and track quality only through the existing offline golden set going forward.
- D Train the LLM judge directly on the thumbs labels, so that user feedback fully replaces the human-labeled calibration set for it.
Answer: B. Raters are a biased sample. Pair thumbs with implicit signals like retries and abandonment, and label a sample of unrated traffic.
The 2% who click thumbs are self-selected, so their rate does not represent the 98% who stay silent, and thumbs-down examples overrepresent a few failure types. Combining explicit feedback with **implicit signals and labeled random samples** costs review effort but gives an unbiased quality estimate. Dropping feedback (C) throws away the earliest real-world failure signal. Thumbs alone are adequate only when rating rates are high and raters resemble the full user base.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?