Agents And Tool Use hard
Users report that a scheduling agent occasionally books the wrong meeting room, yet its final messages name the correct room. Logs capture only the user's request and the final response. Which instrumentation change best enables root-cause analysis?
- A Log the final response plus a rationale the model writes at the end describing which steps it took and why it chose them.
- B Raise log verbosity on the booking API so every request it receives, with full parameters, is captured for later review.
- C Store full transcripts only for runs that end in an error, and sample a small fraction of the remaining runs to keep storage costs under control.
- D Record a trace per run with a span for each model and tool call: inputs, exact emitted arguments, raw results, latency, tokens.
Answer: D. Record a trace per run with a span for each model and tool call: inputs, exact emitted arguments, raw results, latency, tokens.
**Traces** of every step show where the model's emitted arguments diverged from what its final message claims, which is the only place this bug is visible. Full step data costs storage and raises privacy risk, usually managed with retention limits and redaction. Error-only capture misses this failure because wrong bookings succeed; it becomes acceptable once a signal flags bad runs. A post-hoc rationale is generated text, not a record, and API logs lack the model's context.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are defining a 'schedule_meeting' tool for an agent. Which schema decision most reduces wrong or malformed calls?
- A model emits three parallel tool calls: `get_inventory`, `get_price`, and `get_shipping`. To cut latency, the host appends each result to the transcript as it arrives and immediately re-invokes the model. Runs start failing with malformed-request errors, and some produce hallucinated shipping costs. What should the host do?
- A team splits a coding agent into planner, coder, tester, and reviewer agents, and each receives the full transcript of the previous agent. The expected quality gains did not appear. Cost tripled, and the reviewer often fixates on approaches that earlier agents abandoned. When is a multi-agent split actually justified?
- A support agent should remember each customer's preferences across months of sessions. The team appends every past session summary to the system prompt. After six months, prompts exceed 80,000 tokens, latency has doubled, and the agent sometimes ships to an address the customer replaced weeks ago. What redesign fixes both problems?
- A teammate proposes splitting a single coding agent into five agents, arguing that more agents means smarter results. When is a multi-agent split actually justified?
- A data-analysis agent runs model-generated Python inside the same container as your API service. That container has database admin credentials in environment variables. A security review flags this as critical. What is the best mitigation?