All Questions
Llm Gateway Cost And Caching medium

Your gateway applies one 10-second total timeout to every call. Long-form streamed chat answers get cut off mid-response. Raising the global timeout to 120 seconds fixes chat, but intent-classification calls to a degraded provider now hang for a minute. What policy fits?

  1. A Raise the global timeout to 120 seconds and rely on client-side cancellation to abort classifier calls that take too long.
  2. B Set per-route timeouts: short total for classification; for streaming, a first-token limit plus an idle-gap limit.
  3. C Keep 10 seconds everywhere but disable streaming, so long responses arrive as one payload that finishes within the limit.
  4. D Set every timeout to the p50 latency of that route and retry immediately on expiry, since most slow calls succeed on retry.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions