Llm Gateway Cost And Caching hard
Your gateway applies one 10-second total timeout to every route. Long streaming answers of about 2,000 tokens get cut off mid-response, while classification calls hang for the full 10 seconds when the provider degrades. What is the right fix?
- A Raise the global timeout to 120 seconds for every route, since a slow request that eventually succeeds costs less than paying for a retry.
- B Use per-route timeouts: for streaming, a time-to-first-token deadline plus an idle-gap limit; for classification, a short total deadline.
- C Disable streaming for long outputs and buffer the full response, so a single total timeout can be applied uniformly across all routes.
- D Keep 10 seconds but on timeout re-request the stream from the last received token, since providers resume generation from a partial output.
Answer: B. Use per-route timeouts: for streaming, a time-to-first-token deadline plus an idle-gap limit; for classification, a short total deadline.
Generation time grows with output length, so a streaming call can legitimately run for tens of seconds. For streaming, the failure signals are **no first token** or **a long gap between tokens**, not total duration. Classification returns only a few tokens, so a short deadline fails fast and leaves time for a fallback. Raising the global timeout (A) fixes streaming but lets degraded classification calls hold workers and users for two minutes.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?