Llm Gateway Cost And Caching medium
To cut cost, your team routed all traffic to the cheapest model. Quality complaints rose, many users regenerate answers, and total spend barely dropped. Which routing policy best balances cost and quality?
- A Keep the cheapest model for all traffic and retry each failed answer up to three times, since small-model retries are cheap.
- B Route everything to the large model, since retries and complaint handling always erase a smaller model's savings.
- C Default to the small model and escalate to the large one on validation failure or a classifier's difficulty signal.
- D Choose the model per request at random weighted by price, so average cost tracks budget and quality averages out.
Answer: C. Default to the small model and escalate to the large one on validation failure or a classifier's difficulty signal.
Routing is a **policy**: the small model handles the easy majority cheaply, and escalation protects the quality floor on hard requests. The tradeoff is added classifier latency and occasional double calls on escalated requests. Blind retries on the small model (A) repeat the same failure and multiply tokens, which is why spend barely dropped. Routing everything to the large model (B) is right only when nearly all traffic is hard.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?