Agents And Tool Use hard
A research agent averages 40 steps per task, and each step appends roughly 3,000 tokens of tool output. Finance estimated cost as 40 times the cost of one call with the initial prompt. Actual spend is more than an order of magnitude higher. What explains the gap, and what is the fix?
- A Output tokens dominate because the model's reasoning lengthens every step; capping max output tokens per step restores the original estimate.
- B Every step resends the growing transcript, so input tokens grow roughly quadratically with steps; clear or compact tool results once used.
- C Per-step cost should be constant, so the overrun points to hidden duplicate retries; adding idempotency keys to tools will close the gap.
- D Prefix caching makes the repeated transcript nearly free, so the gap must come from tool execution; move tools to cheaper compute.
Answer: B. Every step resends the growing transcript, so input tokens grow roughly quadratically with steps; clear or compact tool results once used.
Step k sends the prompt plus k prior tool results, so 40 steps at 3,000 tokens each sum to roughly 2.5 million input tokens instead of tens of thousands. This is **quadratic transcript growth**. Clearing or summarizing stale tool results gives up some recall of early observations in exchange for near-linear cost. Prefix caching discounts the repeated prefix but does not stop context growth or its latency, and output caps only help when outputs, not inputs, dominate.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are defining a 'schedule_meeting' tool for an agent. Which schema decision most reduces wrong or malformed calls?
- A model emits three parallel tool calls: `get_inventory`, `get_price`, and `get_shipping`. To cut latency, the host appends each result to the transcript as it arrives and immediately re-invokes the model. Runs start failing with malformed-request errors, and some produce hallucinated shipping costs. What should the host do?
- A team splits a coding agent into planner, coder, tester, and reviewer agents, and each receives the full transcript of the previous agent. The expected quality gains did not appear. Cost tripled, and the reviewer often fixates on approaches that earlier agents abandoned. When is a multi-agent split actually justified?
- A support agent should remember each customer's preferences across months of sessions. The team appends every past session summary to the system prompt. After six months, prompts exceed 80,000 tokens, latency has doubled, and the agent sometimes ships to an address the customer replaced weeks ago. What redesign fixes both problems?
- A teammate proposes splitting a single coding agent into five agents, arguing that more agents means smarter results. When is a multi-agent split actually justified?
- A data-analysis agent runs model-generated Python inside the same container as your API service. That container has database admin credentials in environment variables. A security review flags this as critical. What is the best mitigation?