Llm Inference Serving easy
You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- A Use much larger batches to maximize tokens per GPU-second, accepting higher latency on each request.
- B Use batch size 1 to minimize per-request latency, so every document finishes as quickly as possible.
- C Rely on speculative decoding as the main lever, since it raises throughput most on saturated GPUs.
- D Keep the chat settings, since the batch size that is best for latency is also best for throughput.
Answer: A. Use much larger batches to maximize tokens per GPU-second, accepting higher latency on each request.
Batch size trades **latency for throughput**. Larger batches reuse each weight read across many sequences, so total tokens per GPU rise even though each request gets slower. That is the right deal when no one is waiting on a single result. Speculative decoding is tempting, but it spends extra compute to cut per-request latency, so it helps at low load and wastes capacity on saturated GPUs. Small batches remain right for interactive chat with tight latency targets.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?
- Speculative decoding with a small draft model cut inter-token latency by about 2x at low traffic. At peak, with about 200 sequences batched per replica, aggregate throughput is 15% lower than with speculation disabled. What explains this, and what is the fix?