Agents And Tool Use medium
A coding agent reads files on most steps, and each read adds about 8K tokens to the transcript. By step 25, each model call costs far more than step 1 did, and late steps dominate the total bill. What explains this, and what is the fix?
- A Each step re-sends the growing transcript, so cost grows roughly quadratically; clear or compact old tool results as you go.
- B Per-token attention cost explodes with length, so the fix is to restart the agent with an empty transcript every few steps.
- C KV cache eviction forces a full recomputation on every step, so the fix is to pin the cache and leave the transcript unchanged.
- D Output tokens dominate as the model explains more each step, so the fix is to cap max output tokens on every model call.
Answer: A. Each step re-sends the growing transcript, so cost grows roughly quadratically; clear or compact old tool results as you go.
Every loop step is a full model call over the whole transcript. With linear growth per step, total input tokens grow **roughly quadratically**. Clearing or summarizing stale tool results keeps each call small, at the risk of dropping details the model later needs. Restarting from empty (B) also cuts cost but throws away task state. It only fits tasks that split cleanly into independent phases.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are defining a 'schedule_meeting' tool for an agent. Which schema decision most reduces wrong or malformed calls?
- A model emits three parallel tool calls: `get_inventory`, `get_price`, and `get_shipping`. To cut latency, the host appends each result to the transcript as it arrives and immediately re-invokes the model. Runs start failing with malformed-request errors, and some produce hallucinated shipping costs. What should the host do?
- A team splits a coding agent into planner, coder, tester, and reviewer agents, and each receives the full transcript of the previous agent. The expected quality gains did not appear. Cost tripled, and the reviewer often fixates on approaches that earlier agents abandoned. When is a multi-agent split actually justified?
- A support agent should remember each customer's preferences across months of sessions. The team appends every past session summary to the system prompt. After six months, prompts exceed 80,000 tokens, latency has doubled, and the agent sometimes ships to an address the customer replaced weeks ago. What redesign fixes both problems?
- A teammate proposes splitting a single coding agent into five agents, arguing that more agents means smarter results. When is a multi-agent split actually justified?
- A data-analysis agent runs model-generated Python inside the same container as your API service. That container has database admin credentials in environment variables. A security review flags this as critical. What is the best mitigation?