All Questions
Llm Gateway Cost And Caching medium

During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?

  1. A Raise retries at both the SDK and gateway layers to five so more requests eventually succeed before users notice.
  2. B Retry on every error, including 400s and content-filter blocks, since provider errors are usually transient in a brownout.
  3. C Retry at one layer only with jittered backoff and a retry budget; fail over or degrade once the budget is spent.
  4. D Remove timeouts on retried calls so each retry waits as long as needed rather than counting as another failure.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions