All Questions
Llm Gateway Cost And Caching hard

Your gateway applies one 10-second total timeout to every route. Long streaming answers of about 2,000 tokens get cut off mid-response, while classification calls hang for the full 10 seconds when the provider degrades. What is the right fix?

  1. A Raise the global timeout to 120 seconds for every route, since a slow request that eventually succeeds costs less than paying for a retry.
  2. B Use per-route timeouts: for streaming, a time-to-first-token deadline plus an idle-gap limit; for classification, a short total deadline.
  3. C Disable streaming for long outputs and buffer the full response, so a single total timeout can be applied uniformly across all routes.
  4. D Keep 10 seconds but on timeout re-request the stream from the last received token, since providers resume generation from a partial output.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions