Llm Gateway Cost And Caching hard
During an outage at your primary provider, the gateway fails an agent feature over to a second provider using the same prompt. Plain chat works, but the agent emits malformed tool calls and loops. What is the root cause, and what does a working fallback require?
- A The secondary model has a smaller context window; truncating conversation history fixes it, because tool-calling behavior is portable across providers.
- B Fallback is transparent at the API layer, so the failures come from cold prompt caches on the second provider and will stop once those caches warm up.
- C Tool schemas and tool-use behavior differ per provider, so fallback needs translated schemas and a prompt evaluated on that model before an outage.
- D The gateway should retry the primary with backoff before failing over, since switching providers mid-conversation corrupts the stored session state.
Answer: C. Tool schemas and tool-use behavior differ per provider, so fallback needs translated schemas and a prompt evaluated on that model before an outage.
Fallback is not transparent. Providers differ in tool definition formats, in how they emit tool calls, and in how they follow the same instructions, so a prompt tuned on one model can break an agent loop on another. A real fallback costs ongoing work: schema translation, a second prompt, and continuous evals. Retrying the primary (D) is right for brief blips, but in a sustained outage it only delays the failure.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?