Llm Gateway Cost And Caching hard
A regional incident halves your provider capacity, and the gateway starts receiving 429s. Traffic is 60% free-tier chat, 25% paid enterprise chat with an SLA, and 15% nightly batch summarization. What should the gateway do?
- A Admit through a priority queue: defer batch and shed free-tier traffic first, so the reduced token capacity is reserved for paid interactive calls.
- B Retry every 429 with jittered exponential backoff at the gateway, since the provider will absorb the retried traffic as its capacity recovers.
- C Apply a uniform 50% throttle to every tenant and tier, which is fair and keeps error rates equal across free, paid, and batch traffic.
- D Route free traffic to the cheapest small model and keep paid traffic on the primary, since cheaper models never share the provider's capacity pool.
Answer: A. Admit through a priority queue: defer batch and shed free-tier traffic first, so the reduced token capacity is reserved for paid interactive calls.
**Priority tiers** exist for this case. When capacity is scarce, deferrable batch work and free traffic absorb the shortage so SLA-bearing paid traffic keeps working, which means accepting a worse free experience to avoid breaching contracts. Retrying every 429 (B) adds load to a provider that is already saturated and slows everyone equally. A uniform throttle (C) is right only when all traffic has the same value and the same obligations.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?