Evals And Observability hard
A support assistant in a regulated domain handles 5M requests per day. Engineers want full prompt and output logs to debug bad answers. Legal requires data minimization for personal data. Trace storage runs about 20 KB per request. Which logging design best balances debuggability, privacy, and cost?
- A Log every prompt and output in full to encrypted cold storage, since encryption at rest satisfies data-minimization requirements on its own.
- B Log only token counts, latency, and status codes for all traffic, and never store any content, so personal data can never reach storage.
- C Log everything in full but restrict log access to on-call engineers, using access control rather than redaction to protect user data.
- D Keep metadata for all requests, and store content for a small sample after PII redaction, with short retention on the content logs.
Answer: D. Keep metadata for all requests, and store content for a small sample after PII redaction, with short retention on the content logs.
**Sampled, redacted content logging** with full metadata keeps aggregate dashboards exact and gives engineers enough real examples to debug. The cost is that some individual failures cannot be replayed. Logging everything with access control (C) feels safe, but it still retains personal data unnecessarily and stores about 100 GB per day of mostly redundant content. Metadata-only logging (B) is the right choice only when content debugging is never needed or is forbidden outright.
One system design question a day, with spaced repetition so the ones you miss come back.
Get the appRelated questions
- In one release, a RAG assistant got a new embedding model and a new chunking strategy, and the end-to-end judge score dropped 6 points. The team only tracks the end-to-end score and has spent a week tweaking the generation prompt with no recovery. What is the best next step?
- Over two months, your team iterated through 40 prompt versions, each tuned against the same 150-case golden set. The offline score rose from 72 to 91. Online task completion and retry rate have not changed. What is the most likely problem, and what is the fix?
- A prompt change produced an impressive answer in a live demo for leadership. The PM wants to ship it today. What should you do?
- You plan to replace human graders with an LLM judge for your answer-quality eval. Before trusting its scores in a CI gate, what should you do first?
- After you swap to a smaller model, p95 latency falls from 900 ms to 350 ms and the ops dashboard is all green. Leadership asks whether answer quality is still fine. What is the right response?
- Your CI gate fails any prompt change that drops the score 3 or more points on a 100-point rubric over 60 cases. Production generates at temperature 0.7. Running the eval twice on the same commit gives scores that differ by up to 4 points, and engineers have started rerunning failed gates until they pass. How do you fix the gate?