Llm Inference Serving hard
A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- A Compute: 256 concurrent decode streams saturate tensor-core FLOPs long before memory matters, so prefill throughput is what caps the session count.
- B Interconnect: tensor-parallel all-reduce traffic grows with session count and saturates the intra-node GPU links well before 256 sessions are reached.
- C KV cache: at roughly 320 KB per token, each 32K session needs about 10 GB, so only around 45 full-context sessions fit beside the weights.
- D Weights: the 140 GB of fp16 weights are the problem, and moving to 4-bit weights frees enough HBM to reach 256 full-context sessions.
Answer: C. KV cache: at roughly 320 KB per token, each 32K session needs about 10 GB, so only around 45 full-context sessions fit beside the weights.
Per-token KV is 2 x 80 layers x 1,024 KV dims x 2 bytes, about 320 KB, so a 32K session is about 10 GB and 256 sessions need roughly 2.7 TB against about 500 GB left after weights. The tempting fix, 4-bit weights, frees only about 100 GB, which buys around 10 more sessions, because **KV cache dominates memory at long context**. Weight quantization is the right lever only when contexts are short and the weights are most of the footprint.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?
- Speculative decoding with a small draft model cut inter-token latency by about 2x at low traffic. At peak, with about 200 sequences batched per replica, aggregate throughput is 15% lower than with speculation disabled. What explains this, and what is the fix?