Building a RAG App

This is the capstone — every concept in this track wired into one working Perplexity-style app: hybrid retrieval, Cohere rerank, Claude/GPT-4 with citations, streaming, query caching, RAGAS in CI, A/B testing two retrievers. The complexity of production RAG is in the integration, not any single component — and the integration is exactly what solo founders turned into billion-dollar products in 2024–2026. ~200 lines of Python, end-to-end.

Learning Objectives

After this lesson, you will be able to:

Architect a production RAG app end-to-end — ingestion, embedding, hybrid retrieval, reranking, generation, citations, evaluation, monitoring
Stitch together LangChain or LlamaIndex + Pinecone/Weaviate + Cohere Rerank + Claude/GPT-4 + RAGAS into a working Perplexity-style application
Apply production patterns: streaming responses, query caching, fallback handling, cost tracking, A/B testing different retrievers
Identify the most common production failure modes and the production hardening that prevents them

Prerequisites

Before You Start

Review

Why

Why This Matters

Why solo developers built billion-dollar products on RAG in 2024-2026 — Perplexity, Glean, Mendable, You.com all started with the same RAG playbook; the complexity is in the integration, not in any single component
How a Perplexity-style search works under the hood — query → web crawl + cached docs → hybrid retrieval → cross-encoder rerank → Claude/GPT-4 with citation prompt → streaming response with footnotes; this lesson builds exactly that
Why production RAG looks nothing like the LangChain quickstart — caching, fallbacks, streaming, monitoring, eval pipelines, A/B testing infrastructure are all required; we'll cover the full stack
Build this → Build a Perplexity-clone in ~200 lines: FastAPI endpoint that takes a query, hybrid-retrieves from a vector DB, reranks, generates an answer with inline citations, streams to the user, logs to RAGAS for offline eval

Try it: Step through the full production RAG stackInteractive

Open the /play/rag-pipeline playground to walk a query end-to-end: preprocessing → hybrid retrieval → rerank → prompt assembly → streamed generation with citations. Flip the "cache hit" toggle to see how Anthropic prompt caching cuts cost 90% on repeated context.

#The Production RAG Architecture

USER QUERY
    │
    ▼
┌────────────────────────────────────┐
│ 1. Query Preprocessing             │
│    - Normalization                 │
│    - PII redaction                 │
│    - Query rewriting (optional)    │
└────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────┐
│ 2. Hybrid Retrieval                │
│    - BM25 search (sparse)          │
│    - Dense vector search           │
│    - RRF fusion                    │
│    - Top-50 candidates             │
└────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────┐
│ 3. Reranker                        │
│    - Cohere Rerank or BGE          │
│    - Top-50 → top-10               │
└────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────┐
│ 4. LLM Generation                  │
│    - System prompt with rules      │
│    - Top-10 chunks as context      │
│    - Streaming response            │
│    - Inline citations              │
└────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────┐
│ 5. Post-processing                 │
│    - Citation validation           │
│    - Refusal detection             │
│    - Cache write                   │
└────────────────────────────────────┘
    │
    ▼
USER STREAMING RESPONSE WITH FOOTNOTES
    │
    └──► Logged to eval pipeline (RAGAS)

Trace a query through every stage of the pipeline above with the interactive walkthrough below.

Loading visualization...

#Step-by-Step Implementation

pythonplayground.py · Pyodide

# Minimal Perplexity-clone in ~150 lines
# Run: pip install fastapi uvicorn openai cohere chromadb sentence-transformers rank-bm25
import os
import time
import asyncio
from typing import AsyncGenerator
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

# --- Mock components (replace with real APIs) ---
import chromadb
from sentence_transformers import SentenceTransformer
from rank_bm25 import BM25Okapi

# Step 1: Build the corpus + index (run once at startup)
embedder = SentenceTransformer('all-MiniLM-L6-v2')
client = chromadb.Client()
collection = client.get_or_create_collection("docs")

corpus = [
  {"id": "1", "title": "RAG Overview", "text": "RAG combines retrieval with LLM generation for grounded answers."},
  {"id": "2", "title": "Hybrid Retrieval", "text": "Hybrid retrieval combines BM25 with dense embeddings via RRF."},
  {"id": "3", "title": "Reranking", "text": "Cross-encoder rerankers like Cohere Rerank lift NDCG@10 by 10-30%."},
  {"id": "4", "title": "Long Context", "text": "Long-context LLMs hit 60% recall on 1M-token inputs; RAG wins on cost."},
  # ... in production: thousands or millions of docs
]

doc_texts = [d["text"] for d in corpus]
doc_embs = embedder.encode(doc_texts, normalize_embeddings=True)

# Add to vector DB
for i, doc in enumerate(corpus):
  collection.add(
      ids=[doc["id"]],
      embeddings=[doc_embs[i].tolist()],
      documents=[doc["text"]],
      metadatas=[{"title": doc["title"]}],
  )

# BM25 index for sparse retrieval
bm25_corpus = [doc["text"].lower().split() for doc in corpus]
bm25 = BM25Okapi(bm25_corpus)

# --- Step 2: Hybrid retrieval with RRF ---
def hybrid_retrieve(query: str, k: int = 50) -> list:
  q_emb = embedder.encode(query, normalize_embeddings=True).tolist()
  dense_results = collection.query(query_embeddings=[q_emb], n_results=k)
  dense_ids = dense_results["ids"][0]

bm25_scores = bm25.get_scores(query.lower().split())
  bm25_top = sorted(range(len(corpus)), key=lambda i: -bm25_scores[i])[:k]
  bm25_ids = [corpus[i]["id"] for i in bm25_top]

# RRF fusion (k=60)
  scores = {}
  for r in [dense_ids, bm25_ids]:
      for rank, doc_id in enumerate(r):
          scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (60 + rank + 1)
  fused = sorted(scores, key=lambda x: -scores[x])
  return [d for d in corpus if d["id"] in fused[:k]]

# --- Step 3: Mock reranker (use real Cohere Rerank in prod) ---
def mock_rerank(query: str, candidates: list, top_n: int = 10) -> list:
  # Replace with: cohere.Client().rerank(query=query, documents=[c["text"] for c in candidates], top_n=top_n)
  # Here we just take the first top_n for demo
  return candidates[:top_n]

# --- Step 4: LLM with citation prompt ---
SYSTEM_PROMPT = """You are a helpful research assistant.
Answer the user's question using ONLY the provided context.
For every claim in your answer, cite the relevant source(s) using [1], [2], etc.
If the answer is not in the context, say "I don't have enough information to answer."
Do not invent information."""

def build_user_prompt(query: str, chunks: list) -> str:
  context = "\n\n".join([f"[{i+1}] {c['title']}: {c['text']}" for i, c in enumerate(chunks)])
  return f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer with citations:"

# Mock streaming LLM (replace with OpenAI/Anthropic streaming API)
async def stream_llm_response(query: str, chunks: list) -> AsyncGenerator[str, None]:
  # Real implementation:
  # async for chunk in openai.AsyncOpenAI().chat.completions.create(
  #     model="gpt-4o-mini",
  #     messages=[{"role": "system", "content": SYSTEM_PROMPT},
  #               {"role": "user", "content": build_user_prompt(query, chunks)}],
  #     stream=True,
  # ):
  #     yield chunk.choices[0].delta.content or ""
  answer = f"Based on the retrieved context, {chunks[0]['text']} [1]\n\nAdditionally, {chunks[1]['text']} [2]" if len(chunks) >= 2 else "Insufficient context."
  for word in answer.split():
      yield word + " "
      await asyncio.sleep(0.02)

# --- Step 5: FastAPI endpoint ---
app = FastAPI()

class Query(BaseModel):
  q: str

@app.post("/search")
async def search(body: Query) -> StreamingResponse:
  candidates = hybrid_retrieve(body.q, k=50)
  reranked = mock_rerank(body.q, candidates, top_n=10)
  return StreamingResponse(stream_llm_response(body.q, reranked), media_type="text/event-stream")

# --- Run with: uvicorn app:app --reload, then ---
# curl -X POST http://localhost:8000/search -d '{"q": "How does hybrid retrieval work?"}'

print("Architecture built. In production, replace mocks with:")
print("  - Pinecone / Weaviate (vector DB)")
print("  - Cohere Rerank or BGE-Reranker-v2")
print("  - GPT-4o-mini, Claude 3.5 Sonnet, or Gemini 2.0 Flash (LLM)")
print("  - Redis or Upstash for query cache")
print("  - Langfuse for observability + RAGAS evaluation")
print("  - Sentry for error tracking")

print("\nCHALLENGE: Add semantic query caching — hash the embedding of incoming queries")
print("and return cached results when similarity > 0.95. Saves 30-40% of LLM calls.")

Tests · Verify the FastAPI endpoint streams a response with [1] [2] citations. Check that hybrid_retrieve returns docs from both BM25 and dense search.

A quick check before the production hardening table — make sure you can spot the right place to spend optimization effort.

Quick check

Your end-to-end Perplexity-clone has 3 second p95 latency. Breakdown: embed 30ms, vector search 25ms, rerank 120ms, LLM TTFT 800ms, LLM full response 2000ms. Where is the highest-leverage latency improvement?

#Production Hardening Checklist

Concern	Solution
Streaming latency	Use SSE (Server-Sent Events); first-token < 1s
Cost overruns	Per-user rate limits, semantic caching, model tiering (mini for cheap queries, premium for hard ones)
Hallucination	Citation requirement in system prompt; post-process to verify cited claims actually appear in retrieved chunks
Refusals	Detect "I don't have information" responses; route to fallback (web search, escalate to human)
Stale corpus	Incremental re-embedding pipeline; track doc updated_at; re-embed on change
PII leakage	Redact PII before indexing; check responses for accidental PII regurgitation
Prompt injection	Sanitize retrieved chunks; treat them as untrusted; never let chunks override system prompt
Cost monitoring	Track tokens/query and $/query in real-time; alert on anomalies
A/B testing	Run two retrievers in parallel for 5% of traffic; compare RAGAS metrics
Eval CI	RAGAS golden-set runs on every PR; fail builds on > 3% regression

#The Cost Math: Unit Economics of a RAG App

Before you scale, model your unit economics. The playground below computes per-query cost across the realistic 2026 component prices (embed, retrieval, rerank, LLM input, LLM output), shows the cache-hit-rate lever, and finds the volume where self-hosted infra beats hosted APIs.

A Pyodide-backed scratchpad for math lessons.

Loading visualization...

#Common Production Failure Modes

#Hands-On Walkthrough: Building a Perplexity-Clone

The CodePlayground above gives the bones. To productionize:

Replace mocks with real services:
- Vector DB: Pinecone / Weaviate / Qdrant
- Reranker: Cohere Rerank API or BGE-Reranker-v2 self-hosted
- LLM: Claude 3.5 Sonnet (citations API) / GPT-4o-mini / Gemini 2.0 Flash
- Observability: Langfuse / Arize Phoenix / LangSmith
Add safety layers:
- Input sanitization
- Output PII redaction
- Refusal detection
- Citation validation (do the cited chunk ids actually contain the claim?)
Add streaming: replace mock stream_llm_response with real OpenAI / Anthropic streaming SSE.
Add caching:
- Exact-match cache (hash query)
- Semantic cache (embedding similarity)
- Per-user invalidation on subscription changes
Add evaluation:
- RAGAS pipeline runs on every deploy with a golden set
- Online metrics: thumbs-up/down, regeneration rate, time-on-page
- Track which retrievers / rerankers / LLMs win in A/B tests
Add monitoring:
- Cost per query (LLM + reranker + retrieval)
- Latency breakdown (retrieval / rerank / LLM gen)
- Error rates and refusal rates
- User satisfaction signals

#Key Takeaways

Production RAG is integration, not invention. The components are well-understood; the hard part is wiring them together with caching, streaming, monitoring, and eval
Stream first-token in < 1s. Users expect immediate feedback; spinners > 5s feel broken
Citations are required for trust. Every claim must trace to a retrieved chunk; bake this into your system prompt
A/B test retrievers and rerankers. Production data will reveal which configurations win on YOUR queries; don't trust generic benchmarks
Eval pipeline is non-negotiable. RAGAS on a golden set in CI; alert on regressions

#Quick Check

Quick Check1 / 2

Which production hardening provides the largest cost savings for a high-traffic RAG app?

You now have the full production RAG stack — from retrieval through generation through evaluation through deployment. The next track shows how to compose RAG with tool use and planning into autonomous agents.

Building a RAG App

Learning Objectives

After this lesson, you will be able to:

Architect a production RAG app end-to-end — ingestion, embedding, hybrid retrieval, reranking, generation, citations, evaluation, monitoring
Stitch together LangChain or LlamaIndex + Pinecone/Weaviate + Cohere Rerank + Claude/GPT-4 + RAGAS into a working Perplexity-style application
Apply production patterns: streaming responses, query caching, fallback handling, cost tracking, A/B testing different retrievers
Identify the most common production failure modes and the production hardening that prevents them

Prerequisites

Before You Start

Review

Why

Why This Matters

Why solo developers built billion-dollar products on RAG in 2024-2026 — Perplexity, Glean, Mendable, You.com all started with the same RAG playbook; the complexity is in the integration, not in any single component
How a Perplexity-style search works under the hood — query → web crawl + cached docs → hybrid retrieval → cross-encoder rerank → Claude/GPT-4 with citation prompt → streaming response with footnotes; this lesson builds exactly that
Why production RAG looks nothing like the LangChain quickstart — caching, fallbacks, streaming, monitoring, eval pipelines, A/B testing infrastructure are all required; we'll cover the full stack
Build this → Build a Perplexity-clone in ~200 lines: FastAPI endpoint that takes a query, hybrid-retrieves from a vector DB, reranks, generates an answer with inline citations, streams to the user, logs to RAGAS for offline eval

Try it: Step through the full production RAG stackInteractive

#The Production RAG Architecture

USER QUERY
    │
    ▼
┌────────────────────────────────────┐
│ 1. Query Preprocessing             │
│    - Normalization                 │
│    - PII redaction                 │
│    - Query rewriting (optional)    │
└────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────┐
│ 2. Hybrid Retrieval                │
│    - BM25 search (sparse)          │
│    - Dense vector search           │
│    - RRF fusion                    │
│    - Top-50 candidates             │
└────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────┐
│ 3. Reranker                        │
│    - Cohere Rerank or BGE          │
│    - Top-50 → top-10               │
└────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────┐
│ 4. LLM Generation                  │
│    - System prompt with rules      │
│    - Top-10 chunks as context      │
│    - Streaming response            │
│    - Inline citations              │
└────────────────────────────────────┘
    │
    ▼
┌────────────────────────────────────┐
│ 5. Post-processing                 │
│    - Citation validation           │
│    - Refusal detection             │
│    - Cache write                   │
└────────────────────────────────────┘
    │
    ▼
USER STREAMING RESPONSE WITH FOOTNOTES
    │
    └──► Logged to eval pipeline (RAGAS)

Trace a query through every stage of the pipeline above with the interactive walkthrough below.

Loading visualization...

#Step-by-Step Implementation

pythonplayground.py · Pyodide

# --- Mock components (replace with real APIs) ---
import chromadb
from sentence_transformers import SentenceTransformer
from rank_bm25 import BM25Okapi

# Step 1: Build the corpus + index (run once at startup)
embedder = SentenceTransformer('all-MiniLM-L6-v2')
client = chromadb.Client()
collection = client.get_or_create_collection("docs")

doc_texts = [d["text"] for d in corpus]
doc_embs = embedder.encode(doc_texts, normalize_embeddings=True)

# BM25 index for sparse retrieval
bm25_corpus = [doc["text"].lower().split() for doc in corpus]
bm25 = BM25Okapi(bm25_corpus)

bm25_scores = bm25.get_scores(query.lower().split())
  bm25_top = sorted(range(len(corpus)), key=lambda i: -bm25_scores[i])[:k]
  bm25_ids = [corpus[i]["id"] for i in bm25_top]

# --- Step 5: FastAPI endpoint ---
app = FastAPI()

class Query(BaseModel):
  q: str

# --- Run with: uvicorn app:app --reload, then ---
# curl -X POST http://localhost:8000/search -d '{"q": "How does hybrid retrieval work?"}'

print("\nCHALLENGE: Add semantic query caching — hash the embedding of incoming queries")
print("and return cached results when similarity > 0.95. Saves 30-40% of LLM calls.")

Tests · Verify the FastAPI endpoint streams a response with [1] [2] citations. Check that hybrid_retrieve returns docs from both BM25 and dense search.

A quick check before the production hardening table — make sure you can spot the right place to spend optimization effort.

Quick check

#Production Hardening Checklist

Concern	Solution
Streaming latency	Use SSE (Server-Sent Events); first-token < 1s
Cost overruns	Per-user rate limits, semantic caching, model tiering (mini for cheap queries, premium for hard ones)
Hallucination	Citation requirement in system prompt; post-process to verify cited claims actually appear in retrieved chunks
Refusals	Detect "I don't have information" responses; route to fallback (web search, escalate to human)
Stale corpus	Incremental re-embedding pipeline; track doc updated_at; re-embed on change
PII leakage	Redact PII before indexing; check responses for accidental PII regurgitation
Prompt injection	Sanitize retrieved chunks; treat them as untrusted; never let chunks override system prompt
Cost monitoring	Track tokens/query and $/query in real-time; alert on anomalies
A/B testing	Run two retrievers in parallel for 5% of traffic; compare RAGAS metrics
Eval CI	RAGAS golden-set runs on every PR; fail builds on > 3% regression

#The Cost Math: Unit Economics of a RAG App

A Pyodide-backed scratchpad for math lessons.

Loading visualization...

#Common Production Failure Modes

#Hands-On Walkthrough: Building a Perplexity-Clone

The CodePlayground above gives the bones. To productionize:

Replace mocks with real services:
- Vector DB: Pinecone / Weaviate / Qdrant
- Reranker: Cohere Rerank API or BGE-Reranker-v2 self-hosted
- LLM: Claude 3.5 Sonnet (citations API) / GPT-4o-mini / Gemini 2.0 Flash
- Observability: Langfuse / Arize Phoenix / LangSmith
Add safety layers:
- Input sanitization
- Output PII redaction
- Refusal detection
- Citation validation (do the cited chunk ids actually contain the claim?)
Add streaming: replace mock stream_llm_response with real OpenAI / Anthropic streaming SSE.
Add caching:
- Exact-match cache (hash query)
- Semantic cache (embedding similarity)
- Per-user invalidation on subscription changes
Add evaluation:
- RAGAS pipeline runs on every deploy with a golden set
- Online metrics: thumbs-up/down, regeneration rate, time-on-page
- Track which retrievers / rerankers / LLMs win in A/B tests
Add monitoring:
- Cost per query (LLM + reranker + retrieval)
- Latency breakdown (retrieval / rerank / LLM gen)
- Error rates and refusal rates
- User satisfaction signals

#Key Takeaways

Production RAG is integration, not invention. The components are well-understood; the hard part is wiring them together with caching, streaming, monitoring, and eval
Stream first-token in < 1s. Users expect immediate feedback; spinners > 5s feel broken
Citations are required for trust. Every claim must trace to a retrieved chunk; bake this into your system prompt
A/B test retrievers and rerankers. Production data will reveal which configurations win on YOUR queries; don't trust generic benchmarks
Eval pipeline is non-negotiable. RAGAS on a golden set in CI; alert on regressions

#Quick Check

Quick Check1 / 2

Which production hardening provides the largest cost savings for a high-traffic RAG app?

Building a RAG App

After this lesson, you will be able to:

Before You Start

#The Production RAG Architecture

#Step-by-Step Implementation

#Production Hardening Checklist

#The Cost Math: Unit Economics of a RAG App

#Common Production Failure Modes

#Hands-On Walkthrough: Building a Perplexity-Clone

#Key Takeaways

#Quick Check

Deep Dive

Your Reflection

Continue Learning

Discussion

Building a RAG App

After this lesson, you will be able to:

Before You Start

#The Production RAG Architecture

#Step-by-Step Implementation

#Production Hardening Checklist

#The Cost Math: Unit Economics of a RAG App

#Common Production Failure Modes

#Hands-On Walkthrough: Building a Perplexity-Clone

#Key Takeaways

#Quick Check

Deep Dive

Your Reflection

Continue Learning

Discussion