Llm Inference Serving hard
A chat service shows p50 TTFT of 300 ms but p99 TTFT of 8 s during peak hours. Profiling shows prefill for the p99 prompt length takes about 250 ms, and replicas report near-full KV cache occupancy at peak. What is the right first action?
- A Treat it as queueing: add admission control with per-tenant limits, autoscale on queue wait, and shed or defer work once wait exceeds the budget.
- B Switch to a smaller model to cut prefill time, since TTFT is mainly determined by model size and the 8 s tail reflects the cost of prefill.
- C Enable speculative decoding to shorten every request's decode phase, so running sequences finish sooner and the waiting requests start earlier.
- D Increase the maximum batch size so more waiting requests are admitted at once, which empties the queue and brings p99 TTFT close to the p50.
Answer: A. Treat it as queueing: add admission control with per-tenant limits, autoscale on queue wait, and shed or defer work once wait exceeds the budget.
TTFT is **queueing plus prefill**, and with prefill at 250 ms the other 7.7 s is time spent waiting for KV capacity. A smaller model would trim only the 250 ms, and a larger batch cap is useless because KV memory, not the cap, limits admission. Admission control trades rejected or deferred requests for a bounded tail, while shrinking the model is right only when prefill itself dominates TTFT.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?