Llm Gateway Cost And Caching easy
Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A Attach tenant and feature tags to every call and log token counts per request.
- B Split the monthly provider invoice across tenants in proportion to request counts.
- C Give each tenant a separate provider API key and read each key's usage report.
- D Estimate each feature's spend from its share of gateway CPU time and bandwidth.
Answer: A. Attach tenant and feature tags to every call and log token counts per request.
Accurate attribution needs a **tenant and feature tag on every call** plus the input and output token counts, so cost can be computed per request. The tradeoff is the discipline of tagging every call site. Splitting by request count misattributes cost because token sizes vary widely, and per-tenant keys give tenant totals but not a breakdown by feature.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?
- A feature sends 1,500 input tokens and receives 600 output tokens per request. Output tokens cost about 5x input tokens. Which single change cuts cost the most?