Evals And Observability hard
An agent handles a request with a planner LLM call, three tool calls (one sent to a queue worker that makes its own LLM call), and a final generation. In your tracing backend, the worker's LLM spans appear as separate root traces. As a result, per-request token cost and latency are under-reported. What is the right fix?
- A Propagate trace context, such as a W3C traceparent, in the queue message headers so the worker's spans attach to the originating trace.
- B Tag every LLM call with user ID and session ID, then join spans to requests in the warehouse by user and a timestamp window.
- C Raise trace sampling from 10% to 100% so the orphaned worker spans are always captured alongside their parent request spans.
- D Pull total token usage from the provider's billing export each day and divide it by request count to get an accurate per-request cost.
Answer: A. Propagate trace context, such as a W3C traceparent, in the queue message headers so the worker's spans attach to the originating trace.
Traces are only useful if one request links to every model and tool call. Across async boundaries, that requires **explicitly propagating trace context** in message metadata, which costs instrumentation work in every producer and consumer. Joining by user and timestamp (B) works at low concurrency but misattributes spans when one user has overlapping requests. Use it only as a fallback for third-party hops you cannot instrument.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?