All Questions
Llm Gateway Cost And Caching medium

Your FAQ bot uses a semantic cache with a cosine similarity threshold of 0.80. A user asking 'How do I cancel my order?' received the cached answer for 'How do I cancel my subscription?'. What is the right fix?

  1. A Raise the threshold and tune it on labeled query pairs, accepting fewer false hits at the cost of a lower hit rate.
  2. B Lower the threshold so more candidate answers are retrieved, then return whichever cached answer is most recent.
  3. C Replace it with prompt caching, which matches paraphrased questions exactly and so never returns a wrong answer.
  4. D Keep the threshold but scope entries per tenant, since wrong-answer hits usually come from other tenants' cached questions.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions