All Questions
Llm Gateway Cost And Caching hard

An FAQ bot's semantic cache uses a cosine similarity threshold of 0.85 and reaches a 38% hit rate. An audit finds that 'How do I cancel my order?' is being served the cached answer for 'How do I cancel my subscription?'. What is the most sound change?

  1. A Tune the threshold on a labeled set of paraphrases and near-miss pairs, accepting a lower hit rate to cut wrong-intent answers.
  2. B Lower the threshold to raise the hit rate, since the savings from more cache hits outweigh the cost of an occasional mismatched answer.
  3. C Switch from cosine similarity to Euclidean distance on the same normalized embeddings, which better separates short queries differing by one noun.
  4. D Cache only the most frequent queries, since false matches come mostly from long-tail entries that sit in sparse regions of the embedding space.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions