All Questions
Evals And Observability easy

Your CI gate runs a 150-case eval scored on a 100-point rubric. A prompt tweak scores 84 against a baseline of 86, and repeated baseline runs vary by about 1 point. What should the gate do?

  1. A Pass it, because any drop smaller than 10 points is within normal LLM variance and not worth investigating.
  2. B Flag it for review, since a 2-point drop exceeds run-to-run noise and is a common gate threshold.
  3. C Pass it, then turn off the gate for prompt tweaks and run the eval only when the underlying model changes.
  4. D Replace the gate with a 1,000-case public benchmark, since a larger set makes small drops irrelevant to quality.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions