Llm Gateway Cost And Caching hard
An FAQ bot's semantic cache uses a cosine similarity threshold of 0.85 and reaches a 38% hit rate. An audit finds that 'How do I cancel my order?' is being served the cached answer for 'How do I cancel my subscription?'. What is the most sound change?
- A Tune the threshold on a labeled set of paraphrases and near-miss pairs, accepting a lower hit rate to cut wrong-intent answers.
- B Lower the threshold to raise the hit rate, since the savings from more cache hits outweigh the cost of an occasional mismatched answer.
- C Switch from cosine similarity to Euclidean distance on the same normalized embeddings, which better separates short queries differing by one noun.
- D Cache only the most frequent queries, since false matches come mostly from long-tail entries that sit in sparse regions of the embedding space.
Answer: A. Tune the threshold on a labeled set of paraphrases and near-miss pairs, accepting a lower hit rate to cut wrong-intent answers.
The threshold trades **precision against hit rate**. Set it by measurement, using labeled pairs that include near-misses where a single word changes the intent, because a wrong answer costs more than a model call. Switching to Euclidean distance (C) changes nothing: on unit-normalized vectors it ranks results exactly as cosine similarity does. Lowering the threshold (B) is defensible only for low-stakes traffic where any related answer is acceptable.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?