All Questions
Llm Inference Serving hard

Users report that their stream freezes for 1 to 2 seconds mid-response whenever another user on the same replica submits a 30K-token document. The scheduler runs each new request's entire prefill in a single iteration alongside the ongoing decodes. What is the most targeted fix?

  1. A Increase the maximum batch size so the long prefill shares its iteration with more decode sequences and its cost is amortized across all of them.
  2. B Enable speculative decoding so decode sequences emit several tokens per step and can catch up on lost progress once the prefill stall has ended.
  3. C Quantize the KV cache to 8-bit so the 30K-token prefill writes half as many bytes and therefore finishes in roughly half the wall-clock time.
  4. D Split prefill into fixed-size chunks interleaved with decode steps, or route very long prompts to a separate prefill pool of GPUs.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions