Evals And Observability hard
To meet a latency SLO, the team cut p95 latency from 4s to 1.2s by switching to a smaller model and trimming retrieved context from 8 chunks to 3. The ops dashboard is all green: latency, error rate, and timeouts are within target. Leadership asks whether answer quality held. What is the correct answer?
- A It is unknown until the offline eval runs; latency and quality are independent, and both changes can lower answer quality.
- B Quality improved, since faster responses reliably raise user satisfaction and task completion for conversational assistants.
- C Quality held, since the dashboard's error rate and timeout rate did not rise after the model swap and context reduction shipped.
- D Quality held, since fewer retrieved chunks reduce noise in the context window, which consistently raises answer faithfulness.
Answer: A. It is unknown until the offline eval runs; latency and quality are independent, and both changes can lower answer quality.
Operational health metrics (latency, errors, timeouts) say nothing about whether answers are correct. A smaller model and less context trade quality for speed, and only a **quality eval** can measure that tradeoff. Error rate (C) is tempting because it looks like a quality signal, but a fluent wrong answer returns HTTP 200. Trimming context (D) does help when the extra chunks were mostly irrelevant noise, but you have to measure that, not assume it.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?