All Questions
Llm Gateway Cost And Caching hard

A batch pipeline sends a 10,000-token shared instruction prefix with prompt caching enabled on every call. Jobs run about an hour apart, but a cached prefix lasts only a few minutes, so no call ever finds the prefix still cached. What happens to the input cost of the prefix?

  1. A It drops about 90%, because cached prefix tokens are billed at roughly a tenth of the normal input rate on every request that marks them as cacheable.
  2. B It is unchanged, since cache writes are billed at the normal input rate and only cache reads are discounted relative to uncached input.
  3. C It drops modestly, since providers keep frequently used prefixes cached for hours, so hourly jobs still get partial reads on most calls.
  4. D It rises about 25%, since each call pays the cache-write premium and none reads; cluster the jobs within the cache window or disable caching.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions