Llm Inference Serving hard
You switch a 70B-class model from fp16 to 4-bit weight-only quantization. Batch-1 decode became nearly 3x faster, but TTFT on 8K-token prompts barely moved, and at batch 128 the decode speedup shrank to about 1.2x. What explains this?
- A Weight-only quantization cuts bytes read per step, aiding bandwidth-bound decode, but prefill and large-batch decode are compute-bound and still do fp16 math.
- B 4-bit weights force the KV cache to 4-bit as well, which saturates the dequantization units during long prefills and large decode batches.
- C TTFT is dominated by tokenization and network overhead, and at batch 128 the larger KV cache spills to host memory, neither of which depends on weights.
- D Quantization error grows with sequence length and batch size, so the runtime silently falls back to fp16 kernels for long prompts and large batches.
Answer: A. Weight-only quantization cuts bytes read per step, aiding bandwidth-bound decode, but prefill and large-batch decode are compute-bound and still do fp16 math.
Weight-only quantization **shrinks memory traffic, not arithmetic**: weights are dequantized and multiplied at higher precision, so it helps bandwidth-bound small-batch decode and does little for compute-bound prefill or large-batch decode. It also changes quality and kernel behavior, so it needs evaluation and is not free. Weight-and-activation quantization with low-precision matmuls is the variant that also speeds up compute-bound phases.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?