Llm Gateway Cost And Caching hard
Your gateway limits each tenant to 600 requests per minute. One tenant stays under the limit but drives 20x the spend of the others by sending 100,000-token documents and requesting long outputs. Which quota design controls cost?
- A Lower that tenant's request-per-minute limit in proportion to its spend, keeping request count as the single enforcement unit for all tenants.
- B Add a per-tenant cap on concurrent in-flight requests, since long calls hold connections longer and concurrency therefore tracks spend closely.
- C Enforce token quotas with separate input and output limits, reserving max output tokens at admission and reconciling with actual usage.
- D Cap combined input plus output tokens per request at one fixed size for every tenant, since a single total limit prevents expensive calls.
Answer: C. Enforce token quotas with separate input and output limits, reserving max output tokens at admission and reconciling with actual usage.
Spend is driven by **tokens**, not requests, and output tokens cost about 5x input tokens. A combined token cap (D) counts a cheap input token the same as an expensive output token, so a tenant can still spend most of the cap on output. Separate input and output quotas, with output reserved up front from max_tokens, bound cost at admission at the price of extra bookkeeping and some over-reservation. Request-count limits approximate cost only when request sizes are uniform, as with short classification calls.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?