Llm Gateway Cost And Caching medium
The monthly LLM bill tripled, and finance asks which tenant and which feature caused it. The gateway currently logs only the model name and total tokens per day, and all traffic uses one provider API key. What is the right fix?
- A Split the invoice by each tenant's share of HTTP requests through the gateway, which closely approximates their token spend.
- B Rely on the provider's usage dashboard for the API key, since it already records spend by tenant and feature automatically.
- C Require tenant and feature tags on every call and log input, output, and cached tokens per call with the model used.
- D Sample 1% of requests for full logging, since per-call metadata on every request adds too much gateway latency.
Answer: C. Require tenant and feature tags on every call and log input, output, and cached tokens per call with the model used.
**Cost attribution needs a tenant and feature tag on every call**, plus token counts by type, because input, output, and cached tokens are priced very differently. The tradeoff is a bit of logging volume and a required metadata contract for callers, in exchange for answers finance can act on. Splitting by request share (A) breaks badly when a few tenants send long prompts. Sampling (D) works for latency monitoring, but billing requires every call.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?