Evals And Observability medium
Users leave a thumbs-down on about 1% of responses, roughly 3,000 a week. How should the team turn this feedback into a lasting quality improvement?
- A Fine-tune the model weekly on all thumbs-down examples, since explicit negative feedback is a clean training label for the behaviors to avoid.
- B Triage a sample of thumbs-down sessions, label the failure modes, and add representative cases to the golden set used as the change gate.
- C Make the thumbs-down rate the single release gate, since it is user ground truth and makes offline evals redundant for future changes.
- D Set the feedback aside, since users who click thumbs-down are a biased minority whose votes do not reflect the median user's experience.
Answer: B. Triage a sample of thumbs-down sessions, label the failure modes, and add representative cases to the golden set used as the change gate.
A **feedback loop** that turns triaged user failures into golden-set cases means every future change is tested against real problems. The tradeoff is human triage time in exchange for a growing, representative eval set. Using the thumbs-down rate as the only gate (C) is tempting because it is ground truth, but it lags by days and only judges a change after users have seen it. Raw thumbs-down votes are too noisy to use directly as training labels (A) without triage, because the reason and the fix are often unclear.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?