Agents And Tool Use hard
A support agent should remember each customer's preferences across months of sessions. The team appends every past session summary to the system prompt. After six months, prompts exceed 80,000 tokens, latency has doubled, and the agent sometimes ships to an address the customer replaced weeks ago. What redesign fixes both problems?
- A Move to a model with a larger context window so the complete history fits and the latest address is always somewhere in view.
- B Recursively compress the full history into one rolling summary kept in the system prompt, regenerated after every session ends.
- C Fine-tune the model nightly on each customer's sessions so preferences are internalized and no history needs to be in the prompt.
- D Store memories as timestamped records with update semantics in a long-term store, retrieving only those relevant to each request.
Answer: D. Store memories as timestamped records with update semantics in a long-term store, retrieving only those relevant to each request.
**Memory is a retrieval problem.** Discrete records let a new address replace the old one, and retrieval keeps the prompt small. The cost is building write, update, and relevance logic, plus the risk of missing a relevant memory. A rolling summary works for short-lived context, but it is lossy and can blend stale and current facts. A bigger window keeps the contradiction in view and raises cost, and fine-tuning cannot be updated or deleted per customer quickly.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are defining a 'schedule_meeting' tool for an agent. Which schema decision most reduces wrong or malformed calls?
- A model emits three parallel tool calls: `get_inventory`, `get_price`, and `get_shipping`. To cut latency, the host appends each result to the transcript as it arrives and immediately re-invokes the model. Runs start failing with malformed-request errors, and some produce hallucinated shipping costs. What should the host do?
- A team splits a coding agent into planner, coder, tester, and reviewer agents, and each receives the full transcript of the previous agent. The expected quality gains did not appear. Cost tripled, and the reviewer often fixates on approaches that earlier agents abandoned. When is a multi-agent split actually justified?
- A teammate proposes splitting a single coding agent into five agents, arguing that more agents means smarter results. When is a multi-agent split actually justified?
- A data-analysis agent runs model-generated Python inside the same container as your API service. That container has database admin credentials in environment variables. A security review flags this as critical. What is the best mitigation?
- An internal agent has grown to 120 tools, many with overlapping names like search_docs, find_docs, and query_docs. Wrong-tool errors are rising. What is the best next step?