Llm Gateway Cost And Caching hard
For a code-generation feature, your gateway sends every request to a small model that costs about 1/10 as much as the large model, and escalates to the large model when the output fails tests. The small model fails 70% of the time. How should you evaluate this policy?
- A Small-first is still the clear win, since each small call costs a tenth as much as a large call, so the average cost per request drops by roughly 90 percent.
- B Drop escalation and serve only the small model's output, accepting lower quality, since escalation retries are what make cascades expensive.
- C Expected cost is about 0.8x large-only, and 70% of requests pay two calls of latency; routing hard requests straight to the large model is better.
- D Send every request to both models in parallel and keep the better result, which cuts latency and costs the same as escalation on average.
Answer: C. Expected cost is about 0.8x large-only, and 70% of requests pay two calls of latency; routing hard requests straight to the large model is better.
Cascade cost is 0.1 + 0.7 x 1.0 = **0.8x** the large-only cost. That 20% saving is paid for with doubled latency and test runs on most requests, so a classifier or a static route sending this feature to the large model is usually better. Option A ignores that every escalation pays for both calls, and option D costs 1.1x because both models run on every request. Small-first routing does win when the failure rate is low: at 10%, cost falls to about 0.2x.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?