RAG evaluation is the practice of measuring a retrieval-augmented generation pipeline on both of its failure surfaces — did the retriever find the right context, and did the generator use that context faithfully — with separate metrics for each, so a bad answer can be traced to the component that caused it rather than debugged by guesswork.
The reason it needs its own guide is that RAG fails in a way demos never show. The system answers beautifully in the launch review, then quietly degrades: an embedding model update shifts retrieval, a knowledge-base refresh changes chunk boundaries, a prompt tweak makes the generator embellish — and every one of these ships without an error, because the pipeline still returns fluent, confident answers. Eyeballing outputs cannot catch this, and classical NLP scores like BLEU and ROUGE measure surface overlap, not whether the answer was grounded in the retrieved evidence.
This guide covers the full evaluation stack in the order teams should build it: end-to-end outcome evaluation first, retrieval and generation diagnostics second, then dataset strategy, CI/CD gating, production monitoring, and advanced cases such as agentic and multi-turn RAG. For a deeper mathematical treatment of the individual metrics, the companion piece is RAG evaluation metrics: answer relevancy, faithfulness, and more.
What are the two halves of RAG evaluation?
Every RAG pipeline has two distinct failure surfaces: retrieval (finding the right chunks) and generation (using them correctly). Evaluating them separately is the foundational principle of RAG evaluation, because the same bad answer has opposite fixes depending on which half produced it — a retrieval miss means fixing the index, chunking, or embeddings; a generation failure means fixing the prompt or the model.
Teams that only evaluate end-to-end answer quality end up in a specific trap: they see a wrong answer, assume the model is hallucinating, and start prompt-engineering — when the retriever never surfaced the relevant document in the first place. No prompt fixes missing evidence. The inverse trap exists too: tuning retrieval endlessly when the retriever is fine and the generator is ignoring or contradicting what it was given.
The practical rule: end-to-end metrics tell you whether the pipeline is good (and they are what correlates with user outcomes — start there); component metrics tell you where it is failing. You need both, and the component split is what makes error analysis mechanical instead of speculative.
What should you evaluate first in a RAG pipeline?
Start with the outcome the user experiences: whether the final answer correctly and sufficiently addresses the query. Use answer correctness when a trusted reference answer exists, answer relevancy for whether the response serves the request, and one or two product-specific criteria for requirements such as citation quality, refusal behavior, or policy compliance. These end-to-end metrics establish whether the application succeeds before you optimize any component.
Only after that baseline should you add retrieval and generation metrics. Component scores are diagnostic: they explain why the end-to-end result failed and identify which layer to change. Starting with retrieval scores alone can optimize the index without proving that users receive better answers; starting with outcome evaluation preserves the playbook's end-to-end-first principle while still giving engineering the component evidence needed to fix failures.
Retrieval metrics: measuring what your system finds
Retrieval metrics score the chunk list your retriever returns for a query, on two questions: did the relevant chunks make it into the list (recall), and how much of the list is relevant (precision). Everything else — hit rate, ranking metrics — refines these two.
The working set:
- Contextual recall — of all the information needed to answer this query, how much appears in the retrieved chunks? When recall is low, the generator is missing evidence and has less support for a grounded answer, making recall a useful early-warning signal for incomplete or fabricated responses.
- Contextual precision — are the relevant chunks ranked ahead of the irrelevant ones? Precision failures stuff the context window with noise, which both dilutes the generator's attention and, at the margin, induces hallucination from distractor passages.
- Contextual relevancy — what fraction of the retrieved content is actually pertinent to the query at all? A blunt but useful hygiene metric for catching retrieval that has drifted into returning boilerplate or near-duplicates.
- Ranking metrics (MRR, NDCG) — when position matters, mean reciprocal rank rewards putting the single best chunk first (right fit for factoid-style queries with one correct source), while NDCG handles graded relevance across a longer list (right fit for synthesis queries drawing on multiple documents). Most teams need these only after the recall/precision basics are solid.
When to optimize which: if answers are incomplete or fabricated, chase recall. If answers are distracted or diluted — correct information buried in waffle, occasional non-sequiturs from adjacent documents — chase precision. Modern implementations of these metrics use LLM-as-a-judge relevance assessment rather than requiring hand-labeled relevance judgments for every chunk, which is what makes them runnable on real traffic rather than only on annotated benchmarks.
Generation metrics: measuring what your LLM produces
Generation metrics score the answer against two references: the retrieved context (is every claim supported?) and the original query (does it address what was asked?). The first is faithfulness; the second is answer relevancy — and a response can ace one while failing the other, which is exactly why both exist.
Faithfulness: the most critical RAG metric
Faithfulness measures the fraction of claims in the generated answer that are supported by the retrieved context. The mechanism that makes it rigorous is claim decomposition: an evaluator breaks the answer into atomic claims, then classifies each as supported, contradicted, or unsupported by the context — so the score reflects evidence, not vibes. The approach was formalized in academic work on automated, reference-free RAG evaluation and has become the standard across serious evaluation tooling.
High-stakes domains such as healthcare, finance, and legal should use stricter faithfulness gates because the cost of an unsupported claim is asymmetric — the healthcare evaluation guide explains the principle. Do not copy a numerical threshold between tools: faithfulness scores vary with claim decomposition, judge model, rubric, and dataset, so calibrate the gate against human-labeled examples from your application.
The diagnostic value of faithfulness doubles when paired with retrieval metrics: low faithfulness with high contextual recall means the generator is embellishing despite good evidence (fix the prompt or model); low faithfulness with low recall means the generator is compensating for missing evidence (fix retrieval — the generator was set up to fail).
Answer relevancy: factual but off-topic is still a failure
Answer relevancy measures whether the response addresses the question that was actually asked. It exists because faithfulness alone has a blind spot: an answer can be perfectly grounded in the retrieved context and still not answer the user's question — the classic case being a system that retrieves adjacent documentation and faithfully summarizes the wrong thing.
The two metrics are deliberately orthogonal. Faithfulness checks answer-versus-context; relevancy checks answer-versus-query. Track both, because their failure combinations diagnose differently: unfaithful but relevant means hallucination; faithful but irrelevant means retrieval drift or query misunderstanding; both failing means the pipeline broke somewhere fundamental.
RAG evaluation metrics at a glance
Metric | Side | What it measures | Production threshold guidance |
|---|---|---|---|
Contextual recall | Retrieval | Whether the needed information was retrieved at all | Calibrate to labeled queries; set stricter gates where omissions are costly |
Contextual precision | Retrieval | Whether relevant chunks rank above irrelevant ones | Calibrate to ranking judgments; investigate drops from baseline |
Contextual relevancy | Retrieval | Fraction of retrieved content pertinent to the query | Hygiene check; investigate sustained drops |
MRR / NDCG | Retrieval | Ranking quality (single-answer / graded multi-doc) | Secondary; adopt after recall and precision are solid |
Faithfulness | Generation | Fraction of answer claims supported by context | Calibrate to human labels; use the strictest gate for high-stakes claims |
Answer relevancy | Generation | Whether the answer addresses the actual query | Calibrate to human labels and current production baseline |
Custom criteria judges | Generation | Your product's specific rules (tone, policy, format) | Per rule; policy violations gate at 100% |
Human review, LLM-as-a-judge, and the hybrid that actually works
There are three ways to score RAG outputs: human review (the gold standard — expensive, slow, unscalable), LLM-as-a-judge (scalable and cheap, but with known systematic biases toward longer answers and earlier-positioned content), and the hybrid that production teams converge on: humans calibrate the judges, then the judges run at scale.
The hybrid works like this. Humans label a sample of outcomes — good answer or bad answer, not scores — and the automated metrics run on the same sample. Disagreements get resolved by fixing the metric (tighter criteria, adjusted threshold, better judge model) until false-positive and false-negative rates are acceptable for the use case. From then on, the calibrated judges score at scale, and humans spot-check drift after material changes and on a recurring cadence. This is the same metric-validation discipline that underpins all LLM evaluation — RAG just applies it per component.
The bias caveat deserves one practical note: judge biases are mostly systematic, which means calibration catches them. A judge that over-rewards verbose answers will disagree with human labels in a consistent direction, and the calibration loop surfaces exactly that pattern. Uncalibrated judges are where the horror stories come from.
Building your evaluation dataset: golden, synthetic, and production
A complete RAG evaluation dataset draws from three sources: hand-curated golden cases (highest quality, highest cost), synthetic cases generated from your document corpus (fast to scale, must be human-validated), and production traces (most realistic, arrive unlabeled). Mature teams run all three, in that order of trust.
The golden set is the regression backbone — 50–100 personally reviewed cases covering common queries in traffic proportion, edge cases, and every confirmed production failure. Synthetic generation bootstraps coverage before launch and fills topical gaps afterward: cases are generated from your actual corpus, so questions target information your retriever should genuinely be able to find — but validate a sample by hand before trusting a synthetic batch, because generation artifacts (questions no user would ask, answers keyed to chunk boundaries) are common enough to matter.
Production traces are the source that keeps the dataset honest over time. Real queries, real retrievals, real failures — curated into the dataset continuously so evaluation tracks what users actually do rather than what the team imagined at launch. Confident AI automates this curation from traces, which is the structural fix for the golden-set staleness that kills most RAG evaluation programs in their second quarter.
RAG evaluation in CI/CD: gating changes before they ship
Every change to a RAG pipeline — prompt edit, embedding model swap, chunking change, retrieval parameter tune, index refresh — should run against the golden dataset before merging, with per-metric thresholds acting as the quality gate. RAG makes this non-negotiable in a way even general LLM apps don't, because RAG changes have cross-component blast radius: a new embedding model changes what gets retrieved for every query in the system.
The gate mechanics are the standard regression-testing discipline — baseline comparison, per-case verdict flips, non-negotiable case tiers, fast PR subsets with full nightly runs — covered in depth in LLM regression testing. The RAG-specific addition is gating both halves: retrieval metrics and generation metrics gate independently, so a change that improves answers while quietly degrading recall (or vice versa) gets caught for what it is. A pipeline that only gates end-to-end scores will pass embedding-model swaps that hollow out retrieval, because the generator papers over the gap — until it doesn't.
Production monitoring: catching degradation before users do
RAG quality degrades in production without any deploy on your side: the knowledge base gets updated, query patterns shift, embedding drift accumulates, documents go stale. Production monitoring runs the same faithfulness, relevancy, and retrieval metrics on live traces — continuously or sampled — and alerts on trend regressions against a rolling baseline, segmented by use case.
The segmentation matters for the same reason it does everywhere in production issue detection: RAG degradation is almost always localized. One document category goes stale; one query type starts missing; the aggregate faithfulness chart barely moves while a specific user segment gets meaningfully worse answers. Segment scores by query category, document source, and pipeline version, and alert on relative regression per segment.
When an alert fires, the trace is the diagnostic unit: drill from the score drop to the affected traces, inspect which chunks were retrieved and what the generator did with them, and classify the failure as retrieval or generation before touching anything. That drill-down path — score, trace, retrieved context, verdict — is what separates monitoring that produces fixes from monitoring that produces meetings. It requires trace instrumentation that captures retrieval spans with their chunks; the data model is covered in What Is LLM Tracing?.
Advanced cases: agentic RAG, multi-turn, and adversarial testing
Three scenarios outgrow the standard playbook. Agentic RAG — where an agent decides when and what to retrieve — adds a decision layer to evaluate: was retrieval invoked when it should have been, with sensible queries? Skipped-retrieval-then-hallucinated is the signature agentic RAG failure, and it is invisible to metrics that only score retrievals that happened. Multi-turn RAG inherits all of conversational evaluation: retrieval must serve the conversation's evolving context, and knowledge retention across turns becomes a first-class metric — the full treatment is in evaluating multi-turn chatbots. Adversarial testing rounds out the suite with the inputs you hope never arrive: prompt injection attempts embedded in retrieved documents (a RAG-specific attack surface — your corpus is untrusted input), queries designed to elicit ungrounded claims, and malformed requests that stress the pipeline's guardrails.
RAG evaluation best practices
- Evaluate components separately, but let end-to-end metrics lead. End-to-end correlates with user outcomes; component metrics localize the fix. Adopt in that order.
- One rubric per dimension. A single judge prompt scoring faithfulness, relevancy, and tone at once produces muddy scores; separate metrics produce separate diagnoses.
- Validate synthetic data before trusting it. Hand-review a sample of every generated batch; synthetic artifacts silently skew baselines.
- Set baselines early and track trends, not points. A 0.85 faithfulness score means little; a 0.85 that was 0.91 three weeks ago means everything.
- Calibrate judges against humans before they gate anything. Uncalibrated LLM judges inherit systematic biases; the calibration loop converts them from liability to leverage.
- Feed production back into the dataset. The evaluation program that compounds is the one where every confirmed production failure becomes a permanent test case.
For product managers: how to know if your RAG system is getting worse
You do not need to read traces to see RAG degradation coming. The leading indicators are visible on a dashboard: faithfulness and relevancy score trends drifting down in a specific use case, retrieval recall dropping after a knowledge-base update, rising refusal or "I don't know" rates, and user complaints clustering around one topic. Any one of these, sustained for more than a few days, is a real signal — the aggregate looking fine is not evidence to the contrary, because degradation localizes.
The operational requirement is access: quality dashboards, per-use-case trend lines, and the ability to review flagged answers without filing an engineering ticket. Confident AI is built so PMs run this loop directly — viewing metric trends, drilling into flagged responses, and annotating failures that feed the next evaluation cycle — which turns "is the AI getting worse?" from a meeting into a lookup.
Frequently Asked Questions
What is the difference between faithfulness and answer relevancy in RAG evaluation?
Faithfulness scores the answer against the retrieved context: is every claim supported by the evidence? Answer relevancy scores the answer against the original query: does it address what was asked? They are orthogonal — an answer can be fully grounded yet off-topic (faithful, irrelevant) or on-topic yet fabricated (relevant, unfaithful) — which is why both run on every RAG evaluation.
How do I evaluate retrieval and generation separately in a RAG pipeline?
Score the retrieved chunk list with retrieval metrics — contextual recall (was the needed information found?), contextual precision (was it ranked above noise?) — and score the generated answer with generation metrics — faithfulness against the context, relevancy against the query. The combination localizes any failure: bad retrieval with faithful generation means fix the index; good retrieval with unfaithful generation means fix the prompt or model.
How do I detect if my RAG application is hallucinating?
Run a faithfulness metric that decomposes each answer into atomic claims and verifies each against the retrieved context — unsupported claims are the hallucination, quantified. Pair it with contextual recall: hallucination clusters where recall is low, because generators fill evidence gaps with plausible fabrication. In production, run faithfulness on live traces and alert on downward trends per use case rather than waiting for user reports.
How do I set up automated RAG evaluation in CI/CD?
Run every pipeline change — prompts, embedding models, chunking, retrieval parameters — against a golden dataset in the CI job, with retrieval and generation metrics gating independently against calibrated thresholds. Fail the check on non-negotiable case failures or pass-rate regression versus baseline. Gating both halves separately matters because RAG changes have cross-component blast radius that end-to-end scores alone will miss.
What faithfulness threshold should I target in production?
There is no universal faithfulness threshold because scores vary across metric implementations, judge models, rubrics, and datasets. Run the current system on clinician- or domain-expert-labeled cases, choose the threshold that meets your tolerated false-negative rate, and make the gate stricter for high-stakes claims in healthcare, finance, or legal workflows. Then track regression from that calibrated baseline.
How do I tell if RAG retrieval quality is degrading over time in production?
Track retrieval metrics — contextual recall, precision, relevancy — on production traces as trends against a rolling baseline, segmented by query category and document source. Degradation localizes: a knowledge-base update or embedding drift typically hurts one segment while the aggregate stays flat. Alert on relative regression per segment, and drill from the alert into the affected traces to see which queries stopped retrieving the right chunks.
Read next
- LLM Regression Testing: How to Gate Every Change with CI/CD Evals — the full gating discipline this guide's CI/CD section builds on.
- AI Production Issue Detection: A Failure Taxonomy and Detection Framework — the monitoring framework for catching RAG degradation among every other production failure mode.
- Best RAG Evaluation Tools — how the evaluation tooling landscape compares on the capabilities this guide describes.