Llm Gateway Cost And Caching hard
A feature sends a 3,000-token prompt and gets 1,200-token answers. Output tokens cost about 5x input tokens. One proposal halves the prompt to 1,500 tokens. Another caps answers at 400 tokens with a terser format. Which saves more, and why?
- A Halving the prompt, since input tokens outnumber output tokens 2.5 to 1 per request and therefore dominate the per-request cost of this feature.
- B Capping output: weighted at 5x, the answer is about two thirds of cost, so the cap saves about 44% versus about 17% for halving the prompt.
- C They save about the same, since providers bill on total tokens processed, so removing 1,500 input or 800 output tokens is roughly equivalent.
- D Halving the prompt, since shorter prompts also produce proportionally shorter outputs, compounding the savings on both sides of the bill.
Answer: B. Capping output: weighted at 5x, the answer is about two thirds of cost, so the cap saves about 44% versus about 17% for halving the prompt.
In input-token units, the request costs 3,000 + 5 x 1,200 = 9,000. Halving the prompt gives 7,500 (17% less), and capping output gives 5,000 (44% less), so **output length is the main cost lever**, traded against answer completeness. Option A counts raw tokens instead of price-weighted tokens. Trimming the prompt matters more when outputs are short, as in classification, and when the prompt is not cacheable.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- Your primary LLM provider is fully down for an hour, and no tested fallback exists for your agent feature. The product also has an FAQ answer feature backed by a semantic cache. What is the best degradation mode?
- Your gateway applies one 5-second timeout to all LLM calls. Classification calls work fine, but long streamed answers are frequently cut off. What is the best fix?
- A multi-tenant B2B support assistant answers questions using each customer's private knowledge base. To raise the semantic cache hit rate from 22% to 35%, an engineer proposes one shared cache across all tenants. What is the main problem?
- During a provider brownout returning 429 and 503 errors, the client SDK retries 3 times and the gateway also retries 3 times, both immediately. Provider load spikes, and the bill grows from partially streamed responses. What should you change?
- Finance asks for LLM spend broken down by tenant and by product feature. Today the gateway only sees the total on the provider's monthly invoice. What should you do?
- A retail assistant uses a semantic cache. Weeks after a store changed its holiday hours, users still receive the old hours from cached answers. What is the right fix?