Speculative decoding with a small draft model cut inter-token latency by about 2x at low traffic. At peak, with about 200 sequences batched per replica, aggregate throughput is 15% lower than with speculation disabled. What explains this, and what is the fix?

1. A The draft model's KV cache doubles memory per sequence, shrinking the batch; the fix is to let the draft model share the target model's KV cache.
2. B Batching shifts the target model's sampling distribution and lowers acceptance; the fix is greedy decoding at peak so more drafts are accepted.
3. C At large batch, decode is compute-bound, so verifying draft tokens that get rejected burns scarce FLOPs; scale back or disable speculation as batch grows.
4. D Speculation only speeds up prefill, so at peak, when prefill dominates, it just adds overhead; the fix is to apply it to prompt processing alone.