Llm Gateway Cost And Caching easy
Your multi-tenant B2B gateway is adding a semantic cache for questions like "What is our refund policy?". Each tenant has different policies. What must the cache lookup be scoped by?
- A The tenant ID, so a lookup only matches entries cached for that same tenant.
- B Only the query embedding, since one shared pool across tenants maximizes hits.
- C The model name, so answers from different models are never mixed in one pool.
- D The user's session ID, so each conversation keeps its own fully isolated cache.
Answer: A. The tenant ID, so a lookup only matches entries cached for that same tenant.
A semantic cache shared across tenants **leaks one tenant's answers to another**, because "our refund policy" embeds almost identically for every company. Scoping by tenant gives up some hit rate in exchange for correctness and isolation. Scoping by session is safe but so narrow that hit rate collapses; a global pool would only be acceptable for truly public, tenant-neutral content.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?