Llm Gateway Cost And Caching hard
For tracing, your gateway now puts a request ID and an ISO timestamp at the top of an 8,000-token system prompt, ahead of the tool definitions. Since the change, the provider's prompt cache hit rate is near zero and input spend has risen sharply. Which fix costs the least?
- A Enable semantic caching on the system prompt so that near-identical prefixes differing only in the timestamp are treated as prompt cache hits.
- B Move the request ID and timestamp after the static system prompt and tool definitions so the leading tokens are byte-identical across calls.
- C Round the timestamp to the nearest minute, since the provider caches any repeated block of text in the prompt regardless of where it appears.
- D Move the request ID to a metadata header but keep the timestamp first, since one short changing field cannot invalidate a multi-thousand-token cached prefix.
Answer: B. Move the request ID and timestamp after the static system prompt and tool definitions so the leading tokens are byte-identical across calls.
Prompt caching is an **exact prefix match**. The first token that differs ends the reusable prefix, so a per-request value at the top makes the whole 8,000-token prompt uncacheable. Putting stable content (system prompt, tools) first and volatile content last costs nothing and restores cache reads at about a tenth of the input price. Rounding the timestamp (C) only turns a miss on every request into a miss every minute, and it relies on the false idea that caching matches any shared text.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?