Llm Inference Serving hard
Your p99 inter-token latency SLO is 50 ms. Load tests on one replica show batch 64 gives 35 ms ITL at 2,000 tokens/s, and batch 128 gives 60 ms ITL at 3,200 tokens/s. Finance wants batch 128 everywhere to cut the GPU count by about 40%. What should you do?
- A Cap the running batch near the largest size that meets the 50 ms SLO, add replicas for the extra load, and keep batch 128 for offline jobs.
- B Run batch 128, because higher throughput drains the queue faster, which lowers end-to-end latency enough to offset the slower per-token step.
- C Run batch 128 with speculative decoding to win back the extra 10 ms, since spare compute at large batch sizes makes speculation nearly free.
- D Run batch 128 on half the replicas and batch 64 on the rest with random routing, so the fleet-wide p99 averages out below the 50 ms target.
Answer: A. Cap the running batch near the largest size that meets the 50 ms SLO, add replicas for the extra load, and keep batch 128 for offline jobs.
Batch size trades **cost per token against per-token latency**: larger batches raise throughput but every sequence waits on a heavier step. Splitting the fleet fails because half of all requests see 60 ms, so the fleet p99 is 60 ms, and speculation fails because large batches leave little spare compute. Batch 128 is the right setting for offline or batch workloads that have no ITL SLO.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?