Llm Inference Serving medium
One GPU pool serves interactive chat with a 1-second p99 TTFT target and an offline job that summarizes millions of documents overnight. When the offline job starts, chat TTFT jumps to 8 seconds. Peak chat traffic uses about 60% of the pool. You cannot add GPUs this quarter, and the two workloads have no isolation or compliance requirement between them. What should you do?
- A Keep a single FIFO queue but raise the batch size so offline work clears faster and chat requests wait less.
- B Rate-limit the offline job to a fixed requests-per-second cap, chosen so total load stays under pool capacity.
- C Split into two fixed pools, each sized for its own peak, so offline load can never reach the chat GPUs.
- D Add priority classes: chat is scheduled first and offline work fills spare capacity, preempted when needed.
Answer: D. Add priority classes: chat is scheduled first and offline work fills spare capacity, preempted when needed.
**Priority scheduling** protects chat TTFT while the offline job absorbs the remaining ~40% of capacity plus off-peak hours, at the cost of a longer offline completion time. Fixed pools (C) strand about 40% of the chat pool at peak and more off-peak, which the fixed GPU budget cannot afford; they are right only when hard isolation is required, for example for compliance or different model versions. A fixed RPS cap (B) ignores that per-request cost varies with tokens and that chat load moves through the day.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?