Llm Gateway Cost And Caching hard
A multi-tenant support assistant shares one semantic cache across all tenants, which raised the hit rate from 15% to 35%. A customer of tenant B asks 'What is your refund window?' and gets tenant A's 30-day policy. Which design is correct?
- A Keep the shared cache but raise the similarity threshold to 0.98 so only near-verbatim queries can match across tenant boundaries.
- B Keep the shared cache but scrub tenant names, emails, and account IDs from responses before storing them, so a shared answer reveals nothing private.
- C Replace semantic caching with prompt caching, since the provider's prefix cache returns stored answers and is already isolated per tenant.
- D Scope cache keys by tenant and by the version of the tenant's knowledge base, accepting a lower hit rate to eliminate cross-tenant answers.
Answer: D. Scope cache keys by tenant and by the version of the tenant's knowledge base, accepting a lower hit rate to eliminate cross-tenant answers.
A semantic cache keys on what the question means, but the correct answer depends on **who is asking**. The same question from two tenants needs two different answers, so the tenant must be part of the cache key, at the cost of a lower hit rate. Raising the threshold (A) is the tempting fix, but a verbatim match is exactly the failure here. A global cache is safe only for content that is the same for every tenant, such as public product documentation.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?