Llm Gateway Cost And Caching hard
Your LLM bill doubled month over month, and finance asks which product feature caused it. Gateway logs record only the tenant ID, and two internal services call the provider directly with a shared API key. What gives reliable attribution from now on?
- A Allocate the bill to features in proportion to each service's request count, which approximates spend closely enough for internal chargeback.
- B Estimate per-feature cost from prompt lengths in application logs, since input tokens dominate the bill for most LLM features.
- C Issue a separate provider API key per tenant, so the provider's usage dashboard reports exact spend per feature without gateway changes.
- D Route all calls through the gateway and require tenant and feature tags on each, logging input, output, and cached tokens per request.
Answer: D. Route all calls through the gateway and require tenant and feature tags on each, logging input, output, and cached tokens per request.
Attribution needs a **tenant and feature tag on every call**, plus token counts split by type, because input, cached input, and output tokens are priced differently. Any call that bypasses the gateway is invisible, so the cost is making the gateway a mandatory path and rejecting untagged calls. Estimating from prompt length (B) misses that output tokens cost about 5x input and often dominate. Per-tenant keys (C) give tenant totals but cannot separate features within a tenant.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?