All Questions
Agents And Tool Use medium

A coding agent reads files on most steps, and each read adds about 8K tokens to the transcript. By step 25, each model call costs far more than step 1 did, and late steps dominate the total bill. What explains this, and what is the fix?

  1. A Each step re-sends the growing transcript, so cost grows roughly quadratically; clear or compact old tool results as you go.
  2. B Per-token attention cost explodes with length, so the fix is to restart the agent with an empty transcript every few steps.
  3. C KV cache eviction forces a full recomputation on every step, so the fix is to pin the cache and leave the transcript unchanged.
  4. D Output tokens dominate as the model explains more each step, so the fix is to cap max output tokens on every model call.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions