Llm Gateway Cost And Caching hard
EU tenants contractually require inference to stay in the EU. Your gateway's failover policy sends traffic to the cheapest healthy endpoint worldwide, and during an EU region outage it routed EU requests to a US endpoint. What is the right design?
- A Failover is acceptable as an emergency exception, since residency rules apply to stored data and not to transient inference requests.
- B Keep global failover but require TLS on every cross-region call, which satisfies residency since the US endpoint never stores plaintext.
- C Make residency a hard constraint applied before cost and health, so EU traffic fails over only within the EU or degrades instead.
- D Restrict only the semantic cache to the EU, since cached responses are stored data, while routing live requests to any healthy region.
Answer: C. Make residency a hard constraint applied before cost and health, so EU traffic fails over only within the EU or degrades instead.
Routing policy has an order: **hard constraints** such as residency remove candidates first, then cost, latency, and health choose among the endpoints that remain. EU tenants get lower availability, so you need several in-region endpoints and a defined degraded mode such as queueing. Option A is the tempting rationalization, but residency contracts generally restrict where data is processed, not only where it is stored. Global failover is right only for tenants without residency terms.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?