Llm Gateway Cost And Caching medium
A low-traffic internal tool sends a 4,000-token stable system prompt with each request, but requests arrive only once every few hours. Prompt cache entries expire after a few minutes of inactivity, writes cost about 1.25x normal input, and reads about 0.1x. Should you enable prompt caching for this tool?
- A No: with gaps longer than the TTL, every call is a cache write at about 1.25x, costing more than no caching.
- B Yes: cached reads cost about one tenth of normal input, so any stable prefix cuts its input cost by roughly 90%.
- C Yes: caching also discounts output tokens, which dominate cost, so the savings hold even at very low request rates.
- D No: prompt caching only matches text shared with other tenants' prompts, and an internal tool has no other tenants.
Answer: A. No: with gaps longer than the TTL, every call is a cache write at about 1.25x, costing more than no caching.
Prompt caching pays off only when the prefix is **reused within the TTL**; otherwise every call is a write at about a 25% premium and no read ever happens. The tradeoff is the write premium paid up front versus the 90% read discount, which needs at least a couple of hits per write to break even. Option B is true for high-traffic features but ignores expiry. Option D misreads the mechanism: caching matches an exact prefix, not text shared across tenants.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?