deskthebuildAI EVALUATION
← All notes · 5 min read

How would you evaluate a RAG system without a perfect answer key?

Suppose a hypothetical team wants to improve its RAG assistant, but most user questions have no single canonical answer. Documents change, experts disagree on wording, and production feedback is sparse. The team can inspect a few demos, yet cannot tell whether a retrieval or prompt change improves the system overall.

The absence of perfect labels does not remove the need for evaluation. It means the team must decompose quality into observable claims and combine several imperfect sources of evidence.

Why the common answer is incomplete

Most candidates answer: “Use another LLM to grade the answers.”

Model-based grading can scale, but a judge may reward fluent responses, share blind spots with the system, or miss domain-specific errors. A single score also hides whether failure came from retrieval, evidence use, reasoning, citation, or refusal behavior. Without calibration against humans and deterministic checks, the judge becomes an unexamined source of labels.

A production evaluation program uses explicit rubrics, component metrics, sampled expert review, and online signals with known limitations.

A production approach

1. Define the decision the evaluation supports

Specify whether the team is choosing a retriever, releasing a prompt, changing chunking, or monitoring drift. Define acceptable behavior by workflow: correct supported answers, appropriate clarification, or refusal when evidence is absent. Evaluation should answer a release question. A generic “quality score” cannot tell an engineer whether a particular change is safe to deploy.

2. Separate retrieval from answer generation

For retrieval, measure whether the candidate set contains evidence needed to answer, where it ranks, and whether unauthorized or stale passages appear. For generation, evaluate whether claims are supported by the supplied evidence, whether citations map to the right passages, and whether the response follows the user’s constraints. This split prevents prompt tuning from hiding an index that never retrieved the relevant source.

3. Build cases from production patterns and incidents

Sample privacy-safe queries across frequent workflows, difficult long-tail requests, languages, document types, and customer tiers. Add every confirmed production failure as a regression case. Include unanswerable and ambiguous questions, not only examples with clean answers. Preserve source snapshots or version identifiers so a changing corpus does not silently change the expected evidence.

4. Label evidence before writing ideal prose

Experts can often identify relevant source passages, required facts, prohibited claims, and acceptable refusal behavior more reliably than they can agree on one perfect response. Store these as structured criteria. Multiple phrasings can then pass if they satisfy the same evidence contract. This reduces evaluation bias toward a particular writing style.

5. Combine deterministic and model-based checks

Use deterministic checks for citation existence, schema validity, required fields, forbidden sources, and exact business rules. Use a model judge for nuanced groundedness, completeness, or relevance with a precise rubric and evidence included. Require the judge to identify supporting passages and reasons, not only return a number. Calibrate thresholds against expert-reviewed samples and revisit that calibration after judge changes.

6. Measure disagreement and uncertainty

Collect more than one judgment for a representative subset and track where humans, deterministic checks, and model judges disagree. High disagreement may indicate an ambiguous rubric rather than a bad answer. Route uncertain or high-impact cases to expert review. Report confidence intervals or sample counts alongside averages so small changes are not presented as meaningful improvements.

7. Use pairwise comparisons for release decisions

When absolute grading is difficult, show evaluators two anonymized outputs with the same evidence and ask which better satisfies the rubric or whether they are equivalent. Randomize order to reduce position bias. Pairwise results are useful for comparing a proposed release with the current baseline, but they still need failure-category analysis because a small overall win can hide a severe safety regression.

8. Connect offline evaluation to production

Canary eligible traffic and monitor user corrections, follow-up questions, citation opens, task completion, support escalation, latency, and cost. These signals are noisy: users may accept a wrong fluent answer or abandon for unrelated reasons. Use them to detect drift and prioritize review, not as ground truth. Feed confirmed failures back into the versioned evaluation set.

Engineering trade-offs

Expert labels are valuable but expensive and slow. Model judges scale cheaply while requiring calibration and monitoring. Source snapshots improve reproducibility but add storage and may retain sensitive content. Pairwise evaluation detects preference but may not reveal whether both answers are unacceptable.

A large static benchmark enables comparison over time but grows stale as products and documents change. Continuously sampled production cases improve relevance while making historical comparison harder. Mature teams usually maintain a stable regression core plus a rotating slice that represents current traffic.

Common mistakes

  • Scoring only the final answer and ignoring retrieval recall.
  • Treating a model judge as objective ground truth.
  • Building a dataset entirely from easy, answerable questions.
  • Using one ideal response as the only acceptable wording.
  • Comparing experiments against different document versions.
  • Reporting an average without segment sizes or failure categories.
  • Optimizing offline scores without checking production task outcomes.

Failure handling and observability

Version the dataset, source snapshot, rubric, judge model, judge prompt, retriever, generator, and thresholds for every run. Store per-case evidence and failure categories so engineers can inspect regressions. Monitor judge disagreement, missing citations, unanswerable-question behavior, retrieval recall, groundedness, latency, and cost by workflow. Alert on distribution drift between the evaluation set and production queries.

If the judge service fails, keep deterministic gates and postpone the release decision rather than defaulting to pass. If source documents change, mark affected cases and rebuild their evidence contract before comparing scores. If offline and online signals conflict, hold wider rollout and review sampled traces. Evaluation uncertainty is a reason to limit blast radius, not a reason to choose the more optimistic metric.

Interview insight

The interviewer is not testing whether you know a fashionable evaluation framework. They are testing whether you can turn ambiguous quality into a decision process with traceable evidence. A staff-level answer decomposes the system, combines imperfect signals, calibrates automated judges, and treats production failures as permanent learning assets.