Ai Coding And Product Systems medium
Your AI code assistant uses one large model for both inline ghost-text completion and the chat panel. Chat users are happy, but completion takes about 1.5 s at p50 and acceptance rates are falling because developers have already typed past the suggestion. What is the best fix?
- A Keep the single large model for both surfaces but enable streaming so the first tokens of each completion appear sooner in the editor.
- B Increase the context window sent with each completion so the large model produces higher-quality suggestions that justify the wait.
- C Route inline completion to a small, fast model with a 200-400 ms budget and keep the large model for the chat surface.
- D Move completion into the same request queue as chat and batch requests more aggressively to raise GPU throughput per suggestion.
Answer: C. Route inline completion to a small, fast model with a 200-400 ms budget and keep the large model for the chat surface.
Inline completion is **latency-bound** (about 200-400 ms end to end), so it trades some suggestion quality for a small model, while chat tolerates seconds and benefits from a larger model. Streaming (A) is tempting but fails because ghost text needs a short, complete suggestion before the next keystroke, and the large model's time to first token alone can blow the budget. Streaming with a large model is the right choice for the chat surface, not for completion.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your assistant uses a classifier to route queries to a small or large model, which saves 60% of GPU spend. Analysis shows about 20% of hard queries are misrouted to the small model, which then gives confident wrong answers, and regenerate clicks on those threads have doubled. What should you do?
- Your workspace AI retrieves the top-k chunks from a shared vector index and passes them to the model with ACL metadata and a system prompt saying 'only use documents this user can access.' A user just received an answer quoting a private HR page they cannot open. What is the correct fix?
- Your support bot answers billing questions. One user has asked the same refund question three times, and retrieval confidence for that topic is low. What should the bot do next?
- Your AI review system receives a 4,000-line pull request spanning 60 files. Sending the whole diff in one prompt yields shallow, generic comments, and real bugs in files near the middle of the diff are missed. How should you structure the review?
- In your ChatGPT-style assistant, power users keep single threads running for weeks, reaching about 150K tokens. Cost per turn keeps climbing, latency is rising, and the model increasingly misses facts from early in the thread. What should you do?
- Your workspace AI filters retrieval using ACLs stored in the vector index, which is rebuilt nightly. An admin removes an employee from a confidential project page, but for the rest of the day the employee can still get AI answers drawn from it. What is the right fix?