All Questions
Llm Gateway Cost And Caching easy

Your team routes all LLM calls through a new central gateway that runs a routing classifier on each request. Median latency for a short autocomplete feature rises noticeably. What is the most likely reason?

  1. A Gateways add no measurable latency, so the provider must have slowed down that week.
  2. B The gateway forces every response to stream, which is slower for short completions.
  3. C The gateway adds a network hop and classifier time before the provider call.
  4. D The gateway's prompt cache misses on every request, which doubles model latency.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions