Llm Gateway Cost And Caching medium
Your gateway limits free-tier tenants to 100 requests per minute. One tenant stays under that limit yet generates about 30x the spend of other tenants with similar request counts. What is the cause and the right control?
- A Their client retries more often, so add idempotency keys at the gateway to deduplicate repeated calls from that tenant.
- B Their traffic misses the prompt cache, so lower their request limit until their cache hit rate matches other tenants.
- C Their requests land on a larger model by default, so pin all free-tier tenants to the smallest model to equalize spend.
- D Request counts ignore size; they send long inputs and outputs, so enforce separate input and output token quotas.
Answer: D. Request counts ignore size; they send long inputs and outputs, so enforce separate input and output token quotas.
**Cost scales with tokens, not requests**: one request with a huge context and a long answer can cost as much as hundreds of short ones. Separate input and output token quotas control spend directly, at the cost of more complex metering and clearer limits to explain to users. Retries (A) would show up as higher request counts, which this tenant does not have. Request-count limits still matter for protecting throughput, but they do not bound cost.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?