All Questions
Llm Gateway Cost And Caching medium

To cut cost, your team routed all traffic to the cheapest model. Quality complaints rose, many users regenerate answers, and total spend barely dropped. Which routing policy best balances cost and quality?

  1. A Keep the cheapest model for all traffic and retry each failed answer up to three times, since small-model retries are cheap.
  2. B Route everything to the large model, since retries and complaint handling always erase a smaller model's savings.
  3. C Default to the small model and escalate to the large one on validation failure or a classifier's difficulty signal.
  4. D Choose the model per request at random weighted by price, so average cost tracks budget and quality averages out.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions