Llm Gateway Cost And Caching medium
Your team moved an autocomplete feature with a 300 ms p95 budget behind the central LLM gateway, which runs a semantic cache lookup and a routing classifier on every request. p95 latency now exceeds the budget. What is the right explanation and response?
- A Gateways are transparent proxies, so the jump must come from the provider and is fixed by switching to a faster provider.
- B The semantic cache lowered the prompt cache hit rate, so every request now pays full prefill time on its input tokens.
- C The added latency is TLS overhead, so terminate TLS at the client edge and send plaintext to the gateway to remove it.
- D Each added hop and model call costs latency; bypass embedding and classification on this route, keeping auth and metering.
Answer: D. Each added hop and model call costs latency; bypass embedding and classification on this route, keeping auth and metering.
**A gateway is not free**: it adds a network hop, and an embedding lookup plus a classifier add model calls before the main request starts. The tradeoff is central control and savings versus latency, so latency-critical routes should keep only cheap steps like auth, quotas, and metering. Option A assumes zero overhead, which the new p95 disproves. Full routing and caching fit chat or batch routes, where hundreds of milliseconds are small relative to generation time.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?