Llm Gateway Cost And Caching medium
Your assistant builds each request as: current timestamp, the user's query, then a 3,000-token system prompt and tool definitions that never change. Provider-side prompt caching is enabled, but the cache hit rate is near zero. What should you change?
- A Add a semantic cache keyed on the full request so prompts that differ only in the timestamp reuse a previous response.
- B Raise the prompt cache TTL, since the provider evicts the cached system prompt before the next matching request arrives.
- C Put the system prompt and tools first and the timestamp and query last, since the cache matches an exact prefix.
- D Send the system prompt as a separate message, since the provider deduplicates identical text blocks anywhere in a request.
Answer: C. Put the system prompt and tools first and the timestamp and query last, since the cache matches an exact prefix.
**Prompt caching is an exact prefix match**: any volatile token at the start breaks the match for everything after it, so stable content (system prompt, tools) must come first and volatile content last. The tradeoff is prompt layout discipline in exchange for cached reads at about one tenth of the input price. Option D fails because caching does not match shared text anywhere in a request. A longer TTL (B) would only help if the prefix were already stable and traffic was sparse.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?