Agents And Tool Use medium
In production, an agent occasionally quotes the wrong refund amount. You only log the user query and the final answer, and engineers cannot reproduce the bug locally. What should you add first?
- A Log the final answer with higher verbosity and add user feedback buttons so wrong answers can be flagged and counted over time.
- B Set temperature to zero so the agent becomes deterministic and any wrong answer can be reproduced locally from the user query.
- C Record a structured trace per run: each model input/output, tool call arguments, tool results, latencies, and token usage.
- D Ask the model to append a natural-language explanation of its reasoning to each final answer, stored alongside the response.
Answer: C. Record a structured trace per run: each model input/output, tool call arguments, tool results, latencies, and token usage.
Agent bugs usually come from an intermediate step, such as wrong tool arguments or a stale tool result. A **step-level trace** is the debugging surface that shows where the run went wrong. Temperature zero (B) is tempting, but live tool results change between runs, so the query alone cannot replay the failure. The tradeoff is storage cost and PII handling for traces, which sampling and redaction help control.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are defining a 'schedule_meeting' tool for an agent. Which schema decision most reduces wrong or malformed calls?
- A model emits three parallel tool calls: `get_inventory`, `get_price`, and `get_shipping`. To cut latency, the host appends each result to the transcript as it arrives and immediately re-invokes the model. Runs start failing with malformed-request errors, and some produce hallucinated shipping costs. What should the host do?
- A team splits a coding agent into planner, coder, tester, and reviewer agents, and each receives the full transcript of the previous agent. The expected quality gains did not appear. Cost tripled, and the reviewer often fixates on approaches that earlier agents abandoned. When is a multi-agent split actually justified?
- A support agent should remember each customer's preferences across months of sessions. The team appends every past session summary to the system prompt. After six months, prompts exceed 80,000 tokens, latency has doubled, and the agent sometimes ships to an address the customer replaced weeks ago. What redesign fixes both problems?
- A teammate proposes splitting a single coding agent into five agents, arguing that more agents means smarter results. When is a multi-agent split actually justified?
- A data-analysis agent runs model-generated Python inside the same container as your API service. That container has database admin credentials in environment variables. A security review flags this as critical. What is the best mitigation?