Llm Inference Serving hard
Your GPU pool scales out when average GPU utilization exceeds 70%. Utilization reads 95% or more even at 20% of peak traffic, so the pool sits at max size overnight. Morning ramps still breach the TTFT SLO because new replicas take about 5 minutes to load weights. What should replace this policy?
- A Scale on tokens generated per second per replica, since output throughput rises linearly with load and reacts instantly to traffic changes.
- B Scale on queue wait time and KV cache occupancy, keeping warm headroom or schedule-based pre-scaling to cover multi-minute cold starts.
- C Scale on GPU memory utilization, since the KV cache fills in proportion to traffic and the memory percentage is a direct measure of load.
- D Keep GPU utilization but raise the threshold to 98% and shorten the cooldown, so scaling fires only on real saturation and reacts faster.
Answer: B. Scale on queue wait time and KV cache occupancy, keeping warm headroom or schedule-based pre-scaling to cover multi-minute cold starts.
GPU utilization reports how often a kernel is running, not how saturated the GPU is, so decode keeps it high even at light load. Memory utilization is the tempting choice, but many runtimes preallocate the KV pool at startup, so it reads nearly flat, and tokens/s plateaus under overload while the queue grows. Queue wait and KV occupancy track real pressure, and **pre-provisioned headroom trades idle GPU cost for SLO safety** given slow cold starts.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?