Ai Coding And Product Systems hard
A ChatGPT-style assistant has power users whose threads reach 120K tokens against a 128K context window. Per-turn cost keeps climbing, and users report the model ignoring constraints they set in the first few messages. What history strategy best fits?
- A Compact older turns into a rolling summary, pin system instructions and user-stated constraints verbatim, and keep recent turns raw.
- B Keep the full transcript in every prompt, since long-context models attend evenly and prefix caching makes re-reading old turns nearly free.
- C Use a sliding window that drops the oldest turns first so the prompt always fits, since recent turns carry nearly all the user's intent.
- D Move the whole thread into a vector store and retrieve the top few chunks per turn, dropping raw recent turns to save the most tokens.
Answer: A. Compact older turns into a rolling summary, pin system instructions and user-stated constraints verbatim, and keep recent turns raw.
Past roughly 50K to 100K tokens, threads need **compaction**. The trade is lossy detail for bounded cost and better attention on what matters, and pinning constraints protects the facts users care about most. A sliding window (C) drops exactly the early constraints that are being ignored, though it is fine for short casual chats. Keeping everything (B) fails because attention to the middle of a long context degrades, and caching does not raise the window limit.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your assistant uses a classifier to route queries to a small or large model, which saves 60% of GPU spend. Analysis shows about 20% of hard queries are misrouted to the small model, which then gives confident wrong answers, and regenerate clicks on those threads have doubled. What should you do?
- Your workspace AI retrieves the top-k chunks from a shared vector index and passes them to the model with ACL metadata and a system prompt saying 'only use documents this user can access.' A user just received an answer quoting a private HR page they cannot open. What is the correct fix?
- Your support bot answers billing questions. One user has asked the same refund question three times, and retrieval confidence for that topic is low. What should the bot do next?
- Your AI review system receives a 4,000-line pull request spanning 60 files. Sending the whole diff in one prompt yields shallow, generic comments, and real bugs in files near the middle of the diff are missed. How should you structure the review?
- In your ChatGPT-style assistant, power users keep single threads running for weeks, reaching about 150K tokens. Cost per turn keeps climbing, latency is rising, and the model increasingly misses facts from early in the thread. What should you do?
- Your workspace AI filters retrieval using ACLs stored in the vector index, which is rebuilt nightly. An admin removes an employee from a confidential project page, but for the rest of the day the employee can still get AI answers drawn from it. What is the right fix?