Llm Gateway Cost And Caching medium
A provider capacity shortage cuts your available throughput to about 60% of normal for an hour. Free and paid requests share one FIFO queue at the gateway, and paid customers are seeing timeouts. What should the gateway do?
- A Rate limit every tenant by request count to 60% of normal, so each tenant absorbs an equal share of the capacity shortfall.
- B Route all overflow traffic to the cheapest available model, since lower quality is acceptable during any capacity incident.
- C Increase retry attempts with exponential backoff for all tenants so requests eventually succeed once capacity returns.
- D Use separate per-tier queues, serve paid traffic first, and shed or degrade free traffic until capacity returns.
Answer: D. Use separate per-tier queues, serve paid traffic first, and shed or degrade free traffic until capacity returns.
**Priority tiers** protect paid traffic during shortages by deciding in advance who gets degraded. The tradeoff is a worse experience for free users in exchange for meeting paid commitments. Equal request-count cuts (A) ignore both tier and token size, so paid users still time out. Retrying everything (C) adds load to an already saturated provider. Equal sharing makes sense only when all tenants have the same contract.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?