Llm Gateway Cost And Caching medium
During a primary provider outage, your gateway fails over to a second provider using the identical prompt and tool definitions. Plain chat works, but the agent's tool calls start failing with malformed arguments. Why?
- A Tool-call formats and prompt behavior differ by provider; fallbacks need mapped schemas and prompts tested on that model.
- B The fallback provider's prompt cache was cold, so the first requests were truncated until the cached prefix was written.
- C The gateway's semantic cache served answers from the primary provider that referenced tool-call IDs the fallback never issued.
- D The fallback's rate limit was hit by the failover surge, and throttled responses come back as partial tool-call JSON.
Answer: A. Tool-call formats and prompt behavior differ by provider; fallbacks need mapped schemas and prompts tested on that model.
**Fallback is not transparent**: providers differ in tool schema formats, argument strictness, and how they follow the same prompt. The tradeoff is ongoing effort to maintain translated schemas and evaluate prompts per fallback model, in exchange for real availability. A cold prompt cache (B) only affects cost and latency, never correctness. A blind fallback is acceptable only for plain text features with no tool use or strict output format.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?