Llm Gateway Cost And Caching hard
An e-commerce assistant semantically caches answers to questions like 'How much is the X200 blender?' with a 24-hour TTL. After a price change, the cache keeps serving the old price for hours. What is the best fix?
- A Tag entries with the versions of the source records used to answer, invalidate on change, and skip caching for highly volatile intents.
- B Lower the similarity threshold so that fresh answers to newly phrased queries displace older entries faster in the nearest-neighbor index.
- C Rely on prompt caching instead, since the provider's prefix cache is refreshed automatically whenever the underlying product catalog changes.
- D Shorten the TTL to one minute across the entire cache, which bounds staleness with no meaningful loss of hit rate on FAQ-style traffic.
Answer: A. Tag entries with the versions of the source records used to answer, invalidate on change, and skip caching for highly volatile intents.
A semantic cache needs a **freshness policy** tied to the data each answer depends on, not just to elapsed time. Version-based invalidation keeps stable answers cached and drops those whose source changed, at the cost of tracking source versions for each entry. A one-minute global TTL (D) does bound staleness, but it wipes out the hit rate for stable FAQ answers. It is reasonable only if most cached content is volatile.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?