All Questions
Llm Gateway Cost And Caching easy

A provider capacity shortage halves your available throughput. Free and paid traffic share one FIFO queue at the gateway, and paid customers are now timing out. What should the gateway do?

  1. A Retry every failed request immediately so the queue drains faster during the shortage.
  2. B Increase the queue length so more requests from all tiers wait instead of failing.
  3. C Route all traffic to the cheapest model so the reduced capacity serves more calls.
  4. D Add priority tiers so paid traffic is served first and free traffic is delayed or shed.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions