All Questions
Guardrails And Safety hard

A finance-ops agent can issue refunds through a tool. Before each refund call executes, an LLM judge reviews it. An injected instruction in a customer email convinced both the agent and the judge to approve a $9,000 refund. What design change best prevents a recurrence?

  1. A Replace the judge with a larger frontier model, since a more capable reviewer is less likely to be fooled by the same text.
  2. B Enforce a fixed amount threshold in the tool server, above which refunds wait for human approval with full context.
  3. C Show the judge only the tool arguments, never the customer email, so injected text cannot reach the reviewer's context.
  4. D Run the judge three times at nonzero temperature and require unanimous approval before the refund tool may execute.

One system design question a day, with spaced repetition so the ones you miss come back.

Get the app

Related questions