Llm Inference Serving hard
Your chat product streams at about 40 tokens/s per user with a median TTFT of 1.5 s. A proposal would spend this quarter's GPU budget on doubling per-stream decode speed to 80 tokens/s by running smaller batches. What is the better decision?
- A Fund the decode speedup, since streaming at 80 tokens/s halves total response time and users perceive faster streams as proportionally faster.
- B Spend the budget on TTFT through queue headroom, prefix caching, and chunked prefill, since 40 tokens/s already exceeds reading speed.
- C Fund the speedup by adding GPUs to each replica, since higher tensor parallelism doubles decode speed without raising the cost per token.
- D Disable streaming and return complete responses, since at 40 tokens/s the stream adds overhead and total latency would drop noticeably.
Answer: B. Spend the budget on TTFT through queue headroom, prefix caching, and chunked prefill, since 40 tokens/s already exceeds reading speed.
People read at about 5 to 10 tokens/s, so streams faster than about 20 tokens/s are not perceived as faster, while a 1.5 s TTFT is the delay users actually feel. Smaller batches would **raise cost per token for an invisible gain**, and more tensor parallelism gives sublinear speedup at higher cost. Raw decode speed is the right investment when the output feeds a machine, such as agent tool chains, where total latency matters more than reading pace.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?