Llm Gateway Cost And Caching easy
A support chatbot builds each prompt in this order: the current timestamp, the user's question, then a 3,000-token system prompt and tool definitions. Prompt caching is enabled, but the cache hit rate is near zero. What is the best fix?
- A Increase the cache TTL so that cached prefixes survive longer between incoming requests.
- B Move the tool definitions into a separate request so they get cached independently.
- C Put the system prompt and tools first, then the timestamp and question.
- D Switch to semantic caching so that prompts with similar wording can reuse the cached system prompt.
Answer: C. Put the system prompt and tools first, then the timestamp and question.
**Prompt caching is an exact prefix match**: the timestamp at the start changes every request, so nothing after it ever matches. The tradeoff is that you must order prompts for cacheability (stable content first, volatile content last). A longer TTL only helps when the prefix is already stable but traffic is sparse, and semantic caching reuses whole answers, not prompt prefixes.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?