Llm Gateway Cost And Caching medium
EU tenants have contracts requiring their data to be processed only in the EU. Your gateway routes by cost and latency, and during an outage it falls back to the cheapest available provider, which runs only in the US. What should the routing policy be?
- A Allow cross-region fallback only during outages, since contracts typically exempt emergency failover from residency.
- B Make residency a hard per-tenant routing constraint with EU-only fallbacks, and degrade rather than leave the region.
- C Strip personal names before a cross-region fallback, since redaction makes any request safe to send to any region.
- D Route EU tenants by lowest latency, since the nearest region will be in the EU and that satisfies residency in practice.
Answer: B. Make residency a hard per-tenant routing constraint with EU-only fallbacks, and degrade rather than leave the region.
**Data residency is a hard constraint**, not a weight to trade against cost: it must filter the candidate list before cost or latency ranking, including for fallbacks. The tradeoff is fewer fallback options and a degraded mode during EU outages, in exchange for staying within the contract. Latency routing (D) usually picks the EU but can still leave the region under load or failure. Cross-region fallback is acceptable only for tenants without residency terms.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?