Llm Gateway Cost And Caching easy
A batch job analyzes documents one at a time: each request has a unique 20,000-token document and a 300-token instruction, and no document is ever sent twice. Should you mark the document for prompt caching?
- A Yes, since cached tokens cost about one tenth of normal input on every call.
- B No, since cache writes cost extra and this prefix is never reread.
- C Yes, since caching the document also reduces the cost of the generated output.
- D No, since prompt caching only applies to prefixes of a few hundred tokens or less.
Answer: B. No, since cache writes cost extra and this prefix is never reread.
Cache writes cost about **1.25x** normal input, and reads about 0.1x, so caching pays off only when a prefix is reused. A prefix that is written once and never read just adds the 25% premium. Option A would be right if the same document were queried several times, for example with multiple questions per document.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?