Llm Inference Serving medium
You serve a 7B-parameter model in fp16 (about 14 GB of weights) on one GPU with roughly 2 TB/s of memory bandwidth and hundreds of TFLOPs of compute. A product manager expects a single user's stream to exceed 1,000 tokens/sec because each token needs only about 14 GFLOPs of math. What single-sequence decode rate should you plan around?
- A Over 1,000 tokens/sec, because a decode step needs about 14 GFLOPs and the GPU's compute covers that many times per second.
- B Roughly 100 to 150 tokens/sec at best, because every decode step must stream all 14 GB of weights from HBM.
- C Roughly 100 to 150 tokens/sec, but adding a second replica behind the load balancer would double this user's stream rate.
- D Under 20 tokens/sec, because reading the KV cache costs more than reading the weights even for short prompts.
Answer: B. Roughly 100 to 150 tokens/sec at best, because every decode step must stream all 14 GB of weights from HBM.
At batch size 1, **decode is memory-bandwidth-bound**: each step reads every weight once, so the ceiling is about 2 TB/s divided by 14 GB, or roughly 140 steps/sec before overhead. The FLOPs estimate (A) holds only for compute-bound work such as prefill or very large batches, where one weight read is shared across many sequences. Extra replicas (C) raise aggregate throughput but not one stream's speed, and KV reads only dominate at long contexts (D).
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?