Llm Gateway Cost And Caching medium
During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- A Raise retries at both the SDK and gateway layers to five so more requests eventually succeed before users notice.
- B Retry on every error, including 400s and content-filter blocks, since provider errors are usually transient in a brownout.
- C Retry at one layer only with jittered backoff and a retry budget; fail over or degrade once the budget is spent.
- D Remove timeouts on retried calls so each retry waits as long as needed rather than counting as another failure.
Answer: C. Retry at one layer only with jittered backoff and a retry budget; fail over or degrade once the budget is spent.
Stacked retries multiply: 4 attempts at each of two layers means up to 16 calls per user request, which deepens the brownout and bills for wasted partial output. A single retry layer with backoff and a **retry budget** trades a few extra user-visible failures for a provider that can recover. More retries (A) is the classic retry-storm mistake. Immediate retries are reasonable only for rare, isolated transient errors, not during a capacity event.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?
- A feature sends 1,500 input tokens and receives 600 output tokens per request. Output tokens cost about 5x input tokens. Which single change cuts cost the most?