Evals And Observability medium
An agent handles each user request with a planner call, two retrievals, three tool calls, and a final generation. p95 latency spiked and cost per request doubled. Each service logs independently, with no shared ID. What instrumentation should you add?
- A Log the final prompt and output for each request with a timestamp, since the final generation call dominates both end-to-end latency and token cost.
- B Build per-service dashboards for average latency and total token spend, since aggregates are far cheaper to store than per-request traces at scale.
- C Propagate one trace ID per user request with a child span for every model, retrieval, and tool call, recording latency and token counts.
- D Turn on verbose debug logging in every service for a week and join the logs by timestamp proximity to reconstruct each request's call sequence.
Answer: C. Propagate one trace ID per user request with a child span for every model, retrieval, and tool call, recording latency and token counts.
**Distributed tracing** links a user request to every model call and tool call, so you can see which span added latency or tokens, such as a retry loop in the planner. The cost is a few to tens of kilobytes of trace storage per request. Per-service aggregates (B) are cheaper, but they cannot show that one request fanned out into extra calls. Joining logs by timestamp (D) breaks under concurrency, which is exactly why a shared trace ID is needed.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?