All Questions
Llm Gateway Cost And Caching medium

A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?

  1. A Answers built from one tenant's private documents can be served to another tenant; partition the cache by tenant ID.
  2. B Embedding similarity scores drift across tenants, so a shared index must be rebuilt nightly to keep the threshold calibrated.
  3. C A shared cache is safe because embeddings are one-way, so no tenant's raw text can be recovered from a cached vector entry.
  4. D The higher hit rate will invalidate the provider's prompt cache, raising input token cost more than the semantic cache saves.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions