Llm Gateway Cost And Caching medium
Your gateway applies one 10-second total timeout to every call. Long-form streamed chat answers get cut off mid-response. Raising the global timeout to 120 seconds fixes chat, but intent-classification calls to a degraded provider now hang for a minute. What policy fits?
- A Raise the global timeout to 120 seconds and rely on client-side cancellation to abort classifier calls that take too long.
- B Set per-route timeouts: short total for classification; for streaming, a first-token limit plus an idle-gap limit.
- C Keep 10 seconds everywhere but disable streaming, so long responses arrive as one payload that finishes within the limit.
- D Set every timeout to the p50 latency of that route and retry immediately on expiry, since most slow calls succeed on retry.
Answer: B. Set per-route timeouts: short total for classification; for streaming, a first-token limit plus an idle-gap limit.
**Timeouts must match the call shape**: classification returns a few tokens and should fail fast, while streaming can legitimately run for a minute as long as tokens keep arriving. The tradeoff is more configuration per route in exchange for fast failure without cutting off healthy streams. A single long timeout (A) lets hung calls tie up connections and user wait time. A total timeout alone is fine only for non-streaming routes with bounded output.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?