All Questions
Evals And Observability medium

An agent handles each user request with a planner call, two retrievals, three tool calls, and a final generation. p95 latency spiked and cost per request doubled. Each service logs independently, with no shared ID. What instrumentation should you add?

  1. A Log the final prompt and output for each request with a timestamp, since the final generation call dominates both end-to-end latency and token cost.
  2. B Build per-service dashboards for average latency and total token spend, since aggregates are far cheaper to store than per-request traces at scale.
  3. C Propagate one trace ID per user request with a child span for every model, retrieval, and tool call, recording latency and token counts.
  4. D Turn on verbose debug logging in every service for a week and join the logs by timestamp proximity to reconstruct each request's call sequence.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions