Llm Inference Serving hard
200 tenants each prepend their own 4K-token system prompt. You run 16 replicas with prefix caching enabled, but round-robin routing gives about a 6% prefix cache hit rate. One tenant generates 30% of all traffic. Which routing policy is best?
- A Least-outstanding-requests routing, since evenly balanced queues maximize the chance that every replica's prefix cache stays warm for all of the tenants.
- B Pure consistent hashing on tenant ID, since pinning each tenant to exactly one replica guarantees a near-100% prefix cache hit rate for everyone.
- C Keep round-robin but enlarge each replica's prefix cache budget so that every replica can hold all 200 system prompts in memory at once.
- D Prefix-affinity hashing with bounded load: send each request to the replica holding its prefix unless that replica is over a load cap, then spill over.
Answer: D. Prefix-affinity hashing with bounded load: send each request to the replica holding its prefix unless that replica is over a load cap, then spill over.
Routing on prefix trades **cache locality against load balance**. Pure consistent hashing sends the 30% tenant to one replica that has about 6% of fleet capacity, so its queue explodes, while bounded-load spillover keeps most of the hits and caps the hot spots. Caching every prompt everywhere would need about 200 x 4K tokens x hundreds of KB, which is hundreds of GB per replica, so it works only with a handful of shared prefixes.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your 70B-class model servers each handle about 40 concurrent chats with 4K-token contexts. Product raises the maximum context to 32K tokens, and many users start using it. What is the main capacity consequence?
- A 70B-class model (80 layers, grouped-query attention with 8 KV heads of dimension 128) is served in fp16 on one 8-GPU node with about 640 GB of total HBM. Product wants 256 concurrent sessions, each holding a full 32K-token context. What is the binding constraint, and what does it imply?
- Your server reserves a contiguous KV region sized for max_tokens = 8,192 for every request at admission, but the average sequence ends near 600 tokens. The node reports out-of-KV-memory at about 20 concurrent requests while GPU compute sits at 30%. Which change best raises concurrency without restricting legitimate long outputs?
- You must summarize 10 million documents overnight, and no user is waiting on any single result. How should you configure the serving stack compared with your interactive chat deployment?
- You serve a 7B-parameter model in fp16 (about 14 GB of weights) for a chat app. Users say single-conversation token generation is too slow, and you are choosing new GPU hardware. Which hardware spec most directly increases per-sequence decode speed?
- Your RAG workload sends 8K-token prompts and gets ~150-token answers from a 70B-class model. You plan to move prefill and decode onto separate GPU pools so long prefills stop stalling decode. What new resource must you size carefully?