Llm Gateway Cost And Caching medium
Your FAQ bot uses a semantic cache with a cosine similarity threshold of 0.80. A user asking 'How do I cancel my order?' received the cached answer for 'How do I cancel my subscription?'. What is the right fix?
- A Raise the threshold and tune it on labeled query pairs, accepting fewer false hits at the cost of a lower hit rate.
- B Lower the threshold so more candidate answers are retrieved, then return whichever cached answer is most recent.
- C Replace it with prompt caching, which matches paraphrased questions exactly and so never returns a wrong answer.
- D Keep the threshold but scope entries per tenant, since wrong-answer hits usually come from other tenants' cached questions.
Answer: A. Raise the threshold and tune it on labeled query pairs, accepting fewer false hits at the cost of a lower hit rate.
The **similarity threshold** trades hit rate against false hits: short questions that differ by one key noun can still embed very close together. Tuning on labeled pairs of 'same intent' and 'different intent' sets the threshold where wrong answers become rare. Prompt caching (C) is an exact prefix match that reduces input cost; it never reuses an answer for a paraphrase. A looser threshold is acceptable only when a near-miss answer is harmless.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?