All Questions
Llm Gateway Cost And Caching medium

A provider capacity shortage cuts your available throughput to about 60% of normal for an hour. Free and paid requests share one FIFO queue at the gateway, and paid customers are seeing timeouts. What should the gateway do?

  1. A Rate limit every tenant by request count to 60% of normal, so each tenant absorbs an equal share of the capacity shortfall.
  2. B Route all overflow traffic to the cheapest available model, since lower quality is acceptable during any capacity incident.
  3. C Increase retry attempts with exponential backoff for all tenants so requests eventually succeed once capacity returns.
  4. D Use separate per-tier queues, serve paid traffic first, and shed or degrade free traffic until capacity returns.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions