← All posts

How to Evaluate a RAG Pipeline: Precision, Recall, and Faithfulness

A practical guide to RAG evaluation: the four metrics that actually matter, how to build an eval set without a week of work, and a minimal Python loop you can ship today.

By Tulasinath Reddy · May 28, 2026 · Tags: RAG Evaluation Tutorial Metrics

Most teams shipping RAG to production can answer "is it working?" with a shrug and a thumbs-up demo. They cannot answer "did it get better or worse this week?" That gap is where evaluation lives — and it's the difference between a system you can tune and a system you have to pray for.

This post is a practical guide to evaluating a RAG pipeline. Four metrics that actually matter, how to set up an eval set without spending a week on it, and the tradeoffs of the major frameworks (RAGAS, DeepEval, custom).

The four metrics that matter

You'll see a dozen named metrics in the RAG literature. Four of them carry most of the signal — the rest are slices or aggregations.

1. Context precision

Of the chunks the retriever returned, what fraction were actually relevant to the answer? If your top-K is 8 and only 2 chunks were on-topic, precision is 0.25 — you're spending 75% of your context budget on noise that will either dilute the answer or push out a real citation.

High precision matters most when context windows are tight, when token cost is a constraint, or when re-ranking is happening in front of an expensive LLM.

2. Context recall

Of the chunks that should have been retrieved to answer the question, what fraction actually made it into the top-K? Recall is the metric that exposes "the LLM hallucinated" as "the retriever missed the right chunk." Recall failures are almost always the real cause of confident-but-wrong answers.

Measuring recall requires ground-truth chunks for each question — which is the part teams skip and then wonder why their hallucination rate won't budge.

3. Faithfulness

Of the claims in the generated answer, what fraction are supported by the retrieved context? Faithfulness catches the case where the LLM filled gaps with parametric knowledge ("I just know this") instead of with the retrieved chunks. A score of 1.0 means every claim is grounded; under ~0.85 means the model is freestyling.

This is the metric that maps most directly to user trust. Citations + high faithfulness = a system users will actually read.

4. Answer relevance

Does the generated answer actually address the user's question? Even with perfect retrieval and high faithfulness, the model can produce an on-topic but off-question response ("here's everything about Stripe" when the user asked "how do I refund a charge?"). Answer relevance is the human-in-the-loop sanity check, usually scored by a stronger LLM judge.

How to set up an eval set without spending a week

The blocker for most teams isn't picking metrics — it's building the labeled dataset. Three practical shortcuts:

1. Mine your existing question logs. If you have any traffic, the queries users actually ask are your evaluation gold. 50 real questions beat 500 synthetic ones every time.
2. Generate questions from your corpus. Use an LLM to read each document and write 3 questions whose answer is in that document. The answers and "ground-truth chunks" are derived automatically. RAGAS calls this TestsetGenerator; you can roll your own in ~50 lines.
3. Start small, version it. 100 questions across 3–4 question types (lookup, multi-hop, comparative, summarization) is plenty for the first iteration. Re-run on every change. Track scores in a CSV or sqlite — no need for fancy infra.

Frameworks: RAGAS, DeepEval, custom

For most teams: RAGAS in dev, custom in CI. Use RAGAS to find the right metric definitions and a baseline; once you understand what you're measuring, port the 200 lines you actually use into your own eval script so CI doesn't pull in 80MB of dependencies.

A minimal evaluation loop in Python

import json
from openai import OpenAI

client = OpenAI()
JUDGE = "gpt-4o-mini"

def faithfulness(answer: str, context: list[str]) -> float:
    """1.0 if every claim in `answer` is supported by `context`."""
    prompt = f'''Given the context and the answer, list each factual claim
    in the answer and mark it SUPPORTED or UNSUPPORTED. Reply only as JSON
    {{"claims": [{{"text": "...", "supported": true/false}}, ...]}}.

    Context: {chr(10).join(context)}
    Answer:  {answer}'''
    rsp = client.chat.completions.create(
        model=JUDGE,
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    claims = json.loads(rsp.choices[0].message.content)["claims"]
    return sum(c["supported"] for c in claims) / max(1, len(claims))

# usage
score = faithfulness(
    answer="VertexRAG stores graphs in FalkorDB and was founded in 2024.",
    context=["VertexRAG persists knowledge graphs in FalkorDB, a Redis-based graph store."],
)
print(f"faithfulness: {score:.2f}")  # 0.50 — the founding date is unsupported

That's the whole pattern — judge LLM, structured output, normalize to a score. The same shape works for precision, recall, and answer relevance with different prompts.

The mistake everyone makes once

Optimizing for one metric in isolation. Push precision to 1.0 by retrieving only 1 chunk and recall collapses. Push recall to 1.0 by retrieving 100 chunks and precision goes to dirt and your token bill explodes. Faithfulness becomes "the model says it doesn't know" if you penalize too aggressively.

The metric that actually correlates with user satisfaction is a weighted mix — usually faithfulness × answer_relevance with precision/recall as guardrails, not targets. Track all four; tune the mix to your domain.

Try it on your own content

Our free RAG Evaluation tool gives you a 0–100 readiness score for any block of text in under 30 seconds — entity density, structural quality, coherence, and chunk-level recall potential. It's the fastest way to know whether your content needs work, or whether your retrieval pipeline does.

And if you want to skip the framework selection entirely, the VertexRAG API returns retrieval + scoring metadata on every query — precision and faithfulness signals come back inline with the answer, so your eval loop becomes diff this week's scores against last week's.

Read more on the VertexRAG blog or try the Graph RAG API.