The leap from "works in my notebook" to "serves 10K users/day" kills 70% of RAG projects. Latency tails, stale embeddings, silent hallucinations, runaway token bills, prompt injection from your own documents — here's what actually breaks in production and the specific patterns (caching, guardrails, eval-in-CI, fallbacks, observability) that the Glean / Perplexity / Notion AI teams use to keep their RAG up.
Learning Objectives
After this lesson, you will be able to:
Build a 'report card' for your RAG system using four RAGAS metrics: faithfulness (Is the answer grounded in the retrieved sources?), answer relevancy (Does it address the question?), context precision (Are the retrieved chunks useful?), and context recall (Did retrieval find all the relevant information?)
Set up monitoring, caching (so repeated questions are instant and free), and cost controls to keep your RAG system fast and affordable in production
Spot the ways a RAG system fails silently -- wrong answers with high confidence -- and build guardrails so it says 'I am not sure' instead of making things up
Construct a robust evaluation dataset that covers happy-path queries, multi-hop queries, out-of-corpus queries, and adversarial queries -- and run automated evaluation on a schedule to catch quality regressions before users do
This is the lesson that separates hobbyists from professionals. Building a RAG demo is a weekend project. Running one in production that people trust is engineering. You are about to learn the skills that make that jump possible.
Production RAG requires evaluation at every stage: retrieval quality, generation quality, end-to-end answer quality, and user satisfaction. You need numbers, not vibes.
A RAG system that works perfectly in demos can fail catastrophically in production. Documents change, user queries are unpredictable, edge cases multiply, and silent failures -- wrong answers delivered with confidence -- erode trust. This lesson covers the full production lifecycle: evaluation, monitoring, caching, scaling, and cost control.
You need a labeled test set: questions paired with expected answers and the ground-truth relevant documents.
pythonrunnable cell
1
2
3
4
5
6
7
8
evaluation_set = [
{
"question": "What is the refund policy for enterprise customers?",
"ground_truth_answer": "Enterprise customers get 90-day refund windows.",
"ground_truth_contexts": ["doc_enterprise_terms_chunk_3"],
},
# ... 50-200 labeled examples
]
Try it! Write 5 question-answer pairs about a topic you know well (your favorite hobby, your school subject, a book you read). For each pair, write the ideal answer and which document section it comes from. Congratulations -- you just built an evaluation dataset. This is the exact process used to test production RAG systems.
RAG queries are expensive: embedding API call + vector search + LLM generation. Caching reduces cost and latency dramatically.
Figure
Three cache layers sit in front of the expensive path, each catching what the one above it missed. The exact query cache is a hash lookup: instant, but only fires on character-identical repeats. The semantic cache matches on embedding similarity, so it also catches paraphrases of a question already answered. Below both sits the full retrieval-and-generation pipeline, which only runs when the question is genuinely new. Hit rates climb as you move up; cost falls the same way.
The LLM ignores the retrieved context and generates from its parametric knowledge.
Symptoms: High context recall but low faithfulness, answers that go beyond the context.
Causes: Weak system prompt, high temperature, context too long (lost in the middle).
Mitigations: Stronger "answer ONLY from context" prompts, lower temperature, fewer retrieved chunks.
Tests · Verify Q3 (no answer in corpus) gets high faithfulness for correctly abstaining. Verify Q4 (fabricated email) gets low faithfulness for hallucinating an email address.
RAGAS metrics provide systematic RAG evaluation. Faithfulness (does the answer match the context?), answer relevancy (does it address the question?), context precision (are retrieved docs relevant?), and context recall (were all needed docs found?) cover the key quality dimensions
Caching and cost optimization are essential at scale. Embedding frequently asked questions, caching retrieval results, and using smaller models for simple queries can reduce costs by 10-100x without sacrificing quality
Production RAG systems need guardrails. Handle out-of-scope queries gracefully, detect when retrieved context is insufficient, and implement fallback strategies rather than generating confident but wrong answers
Monitoring must cover the entire pipeline. Track latency, retrieval quality, answer quality, and user feedback at each stage; a degradation in any component cascades through the entire system
What does the RAGAS 'faithfulness' metric measure?
Congratulations -- you have completed the RAG & Knowledge Systems track! You now understand the full journey: from why RAG exists, through embeddings, vector databases, and chunking, to the complete pipeline, advanced patterns, knowledge graphs, and production deployment. You have the foundation to build RAG systems that are not just clever demos but reliable, evaluated, monitored production systems.