All Questions
Llm Gateway Cost And Caching easy

A batch job analyzes documents one at a time: each request has a unique 20,000-token document and a 300-token instruction, and no document is ever sent twice. Should you mark the document for prompt caching?

  1. A Yes, since cached tokens cost about one tenth of normal input on every call.
  2. B No, since cache writes cost extra and this prefix is never reread.
  3. C Yes, since caching the document also reduces the cost of the generated output.
  4. D No, since prompt caching only applies to prefixes of a few hundred tokens or less.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions