Building a Production ML/LLM System (Capstone)

Every component in this track — pipelines, feature stores, serving, observability, governance — solves one problem. This capstone wires them all into a single system that survives real users, real traffic, and real incidents. The gap between a notebook and a production service is 6-12 months of integration work. Here's the blueprint that compresses it.

Learning Objectives

After this lesson, you will be able to:

Architect an end-to-end production ML/LLM system that integrates everything from this track — data pipelines, feature stores, training, serving, monitoring, governance
Apply the production hardening checklist before any ML system reaches real users
Identify the most common production failure modes and the architectural patterns that prevent them
Deploy a real ML system to a cloud platform (Modal/Vertex AI/SageMaker) with full observability and incident response in place

Prerequisites

Before You Start

Review

Why

Why This Matters

Why this capstone matters more than any individual technique -- production ML is integration; the components are well-understood; the hard part is wiring them together with the right safety, observability, and reliability layers
Why the difference between a Jupyter notebook and a production system is 6-12 months of engineering work -- deployment, monitoring, governance, security, cost optimization, incident response — each is its own engineering discipline
Why production ML failure is the #1 source of "AI doesn't deliver business value" complaints -- the model worked in the lab, but the production system failed at handling drift, scale, security, or operational complexity
Build this --> Pick a real problem (e.g., fraud detection, customer-support chatbot, content moderation); deploy it end-to-end with all pieces this track covered; the project that turns this curriculum into your portfolio

┌─────────────────────────────────────────┐ │ INGESTION & FEATURES │ │ Kafka → Flink → Feature Store (Feast) │ │ + Data contracts, monitoring │ └─────────────────────────────────────────┘ │ ┌─────────────▼───────────────────────────┐ │ TRAINING & VERSIONING │ │ Airflow + MLflow + DVC │ │ + W&B tracking, Docker, distributed │ └─────────────────────────────────────────┘ │ ┌─────────────▼───────────────────────────┐ │ REGISTRY & DEPLOYMENT │ │ MLflow registry → Vertex AI / Triton │ │ + Approval gates, blue-green deploy │ └─────────────────────────────────────────┘ │ ┌─────────────▼───────────────────────────┐ │ SERVING │ │ vLLM / Triton multi-model │ │ + Autoscale, spot, MIG, cost monitoring│ └─────────────────────────────────────────┘ │ ┌─────────────▼───────────────────────────┐ │ MONITORING & GOVERNANCE │ │ Evidently / Arize for drift │ │ Langfuse for LLM tracing │ │ EU AI Act audit logging │ │ PagerDuty for incidents │ └─────────────────────────────────────────┘ │ ┌─────────────▼───────────────────────────┐ │ RETRAIN LOOP │ │ Drift detected → trigger pipeline │ │ Eval gate → registry promote │ └─────────────────────────────────────────┘

#	Layer	Concern	Pass criteria
1	Data	Schema validation	Pydantic/Avro contracts at every pipeline boundary
2	Data	Drift detection	Daily KS / PSI on key features, alert > threshold
3	Training	Reproducibility	Image SHA + git SHA + data hash logged per run
4	Training	Eval gates	Pass golden-set thresholds before promotion
5	Deployment	Blue-green / canary	Progressive rollout, automatic rollback on regression
6	Deployment	Rollback < 60s	Tested in chaos drills quarterly
7	Serving	Authentication	OAuth / API keys / IAM; never anonymous
8	Serving	Rate limiting	Per-user and per-tier limits; cost cap per user
9	Serving	Tracing	Distributed tracing (Langfuse / OpenTelemetry) on every request
10	Serving	Health checks	Liveness + readiness probes; not just HTTP 200
11	Monitoring	Performance dashboards	Latency p50/p95/p99, accuracy, cost per request
12	Monitoring	Alerting	PagerDuty for severity-1; Slack for severity-2; ticket for severity-3
13	Security	PII handling	PII redaction in logs; GDPR-compliant data retention
14	Security	Prompt injection defense	Input + output classifiers; tool sandboxing
15	Governance	Audit logging	Every model decision traceable for EU AI Act compliance
16	Governance	Model card	Public spec of capabilities + limits + bias measurements

Research	Production
Maximize accuracy on benchmark	Minimize cost-per-correct-prediction
Run experiments	Run incidents
Notebooks	DAGs + containers + registries
Best-case latency	p95 / p99 latency under load
One-time eval	Continuous eval + drift monitoring
Single user (you)	Many users + many tenants
Failures = paper rejection	Failures = on-call page at 2am

Building a Production ML/LLM System (Capstone)

After this lesson, you will be able to:

Before You Start

Your Reflection

Continue Learning

Discussion

#The Reference Architecture

#The Production Hardening Checklist (16 items)

#Real-World Reference Architectures

#Customer Support Chatbot (Glean / Ada / Forethought style)

#Fraud Detection (Stripe Radar style)

#Content Moderation (TikTok / Meta style)

#Hands-On: Deploy Your Capstone

#The Production Engineering Mindset

#Key Takeaways

#Quick Check

Deep Dive