All Questions
Llm Gateway Cost And Caching hard

A regional incident halves your provider capacity, and the gateway starts receiving 429s. Traffic is 60% free-tier chat, 25% paid enterprise chat with an SLA, and 15% nightly batch summarization. What should the gateway do?

  1. A Admit through a priority queue: defer batch and shed free-tier traffic first, so the reduced token capacity is reserved for paid interactive calls.
  2. B Retry every 429 with jittered exponential backoff at the gateway, since the provider will absorb the retried traffic as its capacity recovers.
  3. C Apply a uniform 50% throttle to every tenant and tier, which is fair and keeps error rates equal across free, paid, and batch traffic.
  4. D Route free traffic to the cheapest small model and keep paid traffic on the primary, since cheaper models never share the provider's capacity pool.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions