Agents And Tool Use medium
A personal assistant agent should remember user preferences across months of sessions. Today it appends every past session to the system prompt. Prompts are now about 150K tokens, latency is high, and the model often ignores facts stated long ago. What should you change?
- A Move to a model with a larger context window so the entire history fits comfortably without any truncation of old sessions.
- B Keep only the most recent N sessions in the prompt and drop older ones, since recent preferences are the most relevant ones.
- C Summarize the whole history into one paragraph rewritten each session, and prepend that paragraph to every system prompt.
- D Extract durable facts into a long-term store and retrieve only the entries relevant to the current turn into the context.
Answer: D. Extract durable facts into a long-term store and retrieve only the entries relevant to the current turn into the context.
Memory is a **retrieval problem**. Store durable facts, then fetch only what is relevant to the current turn, which keeps prompts small and the facts salient. A bigger window (A) keeps the cost and latency and still buries old facts in the middle of the context. The tradeoff is building extraction and retrieval quality. A rolling summary (C) is fine when the memory is small enough to fit in a few hundred tokens.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- You are defining a 'schedule_meeting' tool for an agent. Which schema decision most reduces wrong or malformed calls?
- A model emits three parallel tool calls: `get_inventory`, `get_price`, and `get_shipping`. To cut latency, the host appends each result to the transcript as it arrives and immediately re-invokes the model. Runs start failing with malformed-request errors, and some produce hallucinated shipping costs. What should the host do?
- A team splits a coding agent into planner, coder, tester, and reviewer agents, and each receives the full transcript of the previous agent. The expected quality gains did not appear. Cost tripled, and the reviewer often fixates on approaches that earlier agents abandoned. When is a multi-agent split actually justified?
- A support agent should remember each customer's preferences across months of sessions. The team appends every past session summary to the system prompt. After six months, prompts exceed 80,000 tokens, latency has doubled, and the agent sometimes ships to an address the customer replaced weeks ago. What redesign fixes both problems?
- A teammate proposes splitting a single coding agent into five agents, arguing that more agents means smarter results. When is a multi-agent split actually justified?
- A data-analysis agent runs model-generated Python inside the same container as your API service. That container has database admin credentials in environment variables. A security review flags this as critical. What is the best mitigation?