Llm Gateway Cost And Caching medium
A product-support bot uses a semantic cache with a 30-day TTL. After the pricing page changed, the bot kept quoting old prices for days because cached answers were still being served. What is the best fix?
- A Disable the semantic cache, since any cached LLM answer can go stale and so cached answers are never safe to reuse.
- B Tie each entry to its source documents' versions and invalidate on change, with shorter TTLs for volatile topics.
- C Raise the similarity threshold so only near-exact repeats hit the cache, which keeps stale answers from being returned.
- D Add 'always use the latest pricing' to the system prompt so the model overrides any outdated cached answer it sees.
Answer: B. Tie each entry to its source documents' versions and invalidate on change, with shorter TTLs for volatile topics.
A semantic cache needs a **freshness policy**: invalidating entries when their source content changes keeps the hit rate on stable FAQs while removing stale answers. The tradeoff is tracking which documents each answer used. A stricter threshold (C) changes which queries match but still serves the same stale answer to an exact repeat. Disabling the cache (A) is right only when the content changes faster than queries repeat.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?