Evals And Observability hard
Monthly LLM spend rose 40% while request volume stayed flat and no new models were adopted. Your traces record token counts and latency on every span: retrieval, each generation call, and each tool call. Several teams shipped prompt and agent changes during the month. How do you find the cost driver most efficiently?
- A Alert on average latency per request, since rising LLM cost tracks rising latency closely and will point straight to the slow feature.
- B Compare monthly provider invoices across environments and API keys to see which service's total spend grew the most over the month.
- C Aggregate span token counts by feature, prompt version, and span type to find which component's tokens per request grew.
- D Manually review 20 sampled transcripts, since a close qualitative read finds the cost driver faster than aggregate dashboards do.
Answer: C. Aggregate span token counts by feature, prompt version, and span type to find which component's tokens per request grew.
Flat volume with rising spend means **tokens per request** grew, for example from longer context, extra agent tool loops, or retries. Only span-level attribution by feature and prompt version shows which change caused it, at the cost of storing a few KB of token metadata per trace. Invoice comparison (B) narrows the search to a service but not to the prompt or loop that changed, so it is sufficient only when each feature has its own API key. Latency (A) is a poor proxy because cached or parallel calls can add cost without adding latency.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?