"It looks pretty good" is how RAG systems silently regress. Without a golden eval set and CI-blocking metrics, you can't tell whether last week's chunking tweak helped or wrecked production. RAGAS, LLM-as-judge, citation-grounded faithfulness — these are how Pinecone, LangSmith, Arize, and every serious enterprise RAG team catch regressions before users do. This lesson is how you turn "vibes" into numbers that fail CI.
Try it: Score a RAG run with RAGAS metrics liveInteractive
Open the /play/rag-eval-metrics playground and drag the retrieved-chunk relevance slider. Watch faithfulness, answer relevancy, context precision, and context recall update — and notice how a single hallucinated sentence collapses faithfulness from 0.95 to 0.40.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
Recall@k=#{all relevant chunks}#{relevant chunks in top-k}
NDCG@k=IDCG@k∑i=1klog2(i+1)reli
MRR (Mean Reciprocal Rank): 1 / (rank of first relevant doc). Useful for QA tasks where one good chunk suffices.
A quick check before we move to LLM-as-judge metrics.
Quick check
Your retriever returns relevant docs at ranks 1, 4, 7 for query A and at rank 9 for query B. Total relevant docs are 3 (A) and 1 (B). What's MRR averaged over these two queries?
LLM-as-judge metrics are powerful but not free — Cohere/OpenAI eval calls cost ~$0.01-0.10 per question. The complement: a small golden set (50-200 hand-curated questions) with known ground-truth chunks and answers.
Recipe
Sample real user queries from production logs (anonymized)
For each, label the ground-truth chunks (which docs SHOULD be retrieved)
Write the ideal answer
Run your RAG pipeline; compute recall@k, NDCG@k, faithfulness, answer relevancy
Commit to CI; fail builds on > 3% regression
Golden sets catch the regressions that aggregate metrics miss. Production teams maintain both.
The RAGAS metrics look opinionated but each maps to a clean information-retrieval quantity. The playground below implements faithfulness, answer relevancy, context precision, and context recall from scratch on a tiny golden set so you can see exactly what RAGAS is computing under the hood.
The 2024+ frontier: don't just check if the answer is correct, check if each claim cites a specific retrieved chunk.
Why it matters: regulators (EU AI Act, FDA, FTC) increasingly require traceable AI outputs. Anthropic and OpenAI now expose citation APIs (claims linked to source chunks). Eval must verify these citations.
Citation precision: % of citations that actually support the claim they're attached to. Citation recall: % of claims that are properly cited.
What Do You Think?
Your RAGAS faithfulness is 0.95 but users complain the answers are 'wrong'. What's the most likely diagnosis?
The answer: faithfulness measures groundedness in retrieved context, not factual correctness. If retrieval pulled wrong chunks but the LLM faithfully grounded its answer in those wrong chunks, faithfulness=1.0 but the answer is wrong. Always pair faithfulness with answer relevancy and context precision.
Retrieval and generation eval are distinct. Recall@k / NDCG / MRR for retrieval; faithfulness / answer relevancy for generation
RAGAS is the canonical LLM-as-judge framework. Works without ground-truth answers; production target faithfulness ≥ 0.90
Golden sets catch what aggregates miss. 50-200 hand-curated questions; commit to CI; fail builds on > 3% regression
Faithfulness ≠ correctness. High faithfulness only means the answer is grounded in retrieved chunks; pair with context precision to catch retrieval errors
Citation-grounded eval is the 2026 frontier. Every claim must trace to a retrieved chunk; required for regulated industries