Llm Inference Serving medium
During a traffic spike your inference queue grows several minutes deep. Clients time out at 30 seconds, but GPUs stay at 100% generating responses for requests whose clients have already given up, so goodput falls to near zero. What is the best immediate fix?
- A Bound the queue by estimated wait time and reject with 429 when projected TTFT exceeds the client's deadline.
- B Raise the client timeout to 120 seconds and keep the queue unbounded so every accepted request eventually completes.
- C Increase the maximum batch size so each GPU drains the backlog faster, lowering queue wait for every request.
- D Trigger the autoscaler on queue depth and hold all requests until the new GPU replicas finish loading weights.
Answer: A. Bound the queue by estimated wait time and reject with 429 when projected TTFT exceeds the client's deadline.
**Fail-fast admission control** rejects some requests so the GPUs only do work whose result will be delivered. Longer timeouts (B) make every user wait minutes, and bigger batches (C) add only modest throughput while raising inter-token latency, which cannot clear a multi-minute backlog. New replicas (D) take minutes to load, so held requests still expire. An unbounded queue is acceptable only for offline batch jobs that have no deadline.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?