Llm Gateway Cost And Caching hard
You add a gateway that runs a small-model classifier on every request to choose a route. An autocomplete feature with a 300 ms p99 budget now misses its SLA. Which decision fits the trade-off?
- A Gateway overhead is negligible next to model inference, so the regression must be in the provider and the gateway path can stay unchanged.
- B Put a semantic cache in front of every route instead of the classifier, since an embedding lookup adds no measurable latency to requests.
- C Bypass the gateway for this feature and call the provider directly, since quotas and cost tags cannot be enforced on low-latency paths.
- D Use a static route for this feature to skip the classifier, and deploy the gateway near its callers so it adds only a short network hop.
Answer: D. Use a static route for this feature to skip the classifier, and deploy the gateway near its callers so it adds only a short network hop.
A gateway adds a **network hop plus classification time**, and a classifier that is itself a model call can consume much of a 300 ms budget. A static route removes the classifier from the critical path but keeps quotas, tagging, and fallbacks. Dynamic routing is worth its latency for long generations, where seconds of inference dwarf it. Bypassing the gateway (C) restores latency but gives up attribution and quotas, which cost microseconds, and option A ignores that the classifier is itself an inference call.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?