Llm Gateway Cost And Caching hard
A batch pipeline sends a 10,000-token shared instruction prefix with prompt caching enabled on every call. Jobs run about an hour apart, but a cached prefix lasts only a few minutes, so no call ever finds the prefix still cached. What happens to the input cost of the prefix?
- A It drops about 90%, because cached prefix tokens are billed at roughly a tenth of the normal input rate on every request that marks them as cacheable.
- B It is unchanged, since cache writes are billed at the normal input rate and only cache reads are discounted relative to uncached input.
- C It drops modestly, since providers keep frequently used prefixes cached for hours, so hourly jobs still get partial reads on most calls.
- D It rises about 25%, since each call pays the cache-write premium and none reads; cluster the jobs within the cache window or disable caching.
Answer: D. It rises about 25%, since each call pays the cache-write premium and none reads; cluster the jobs within the cache window or disable caching.
Cache writes cost about **1.25x** normal input and reads about **0.1x**, so caching pays off only if a prefix is read at least once before it expires. One write plus one read costs 1.35x, against 2.0x for two uncached calls, but with no reuse every call pays the 25% premium for nothing. Option A applies the read discount to writes. For interactive traffic, where the same prefix arrives many times a minute, caching is the right default.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?