The pipeline from data to deployed model is where 90% of ML projects die. Not because the model isn't good — because the data is wrong, the features drift, monitoring is missing, and rollback is impossible. Here's the full production stack, in order.
Learning Objectives
After this lesson, you will be able to:
Follow data through every stage of a real ML pipeline: from raw data to deployed model to ongoing monitoring
Spot what can go wrong at each stage and build safeguards against it
Design pipelines that are repeatable, trackable, and automated -- not just notebooks that work once and break forever
This is the foundational lesson for everything that follows in this track. Understanding the full pipeline will make every other topic -- MLOps, feature stores, monitoring -- click into place. You are building a mental map that will guide your entire ML engineering career.
A production ML pipeline is a directed acyclic graph (DAG) of interconnected stages, each with its own inputs, outputs, validation checks, and failure modes. Understanding the full pipeline is what separates ML engineers from ML hobbyists.
Raw data arrives from databases, APIs, event streams, data lakes, third-party vendors, and user uploads. This stage handles:
Data sources
Batch ingestion: periodic pulls from databases or data warehouses (Snowflake, BigQuery, Redshift)
Streaming ingestion: real-time events via Kafka, Kinesis, or Pub/Sub
External APIs: third-party data enrichment
User-generated: uploads, annotations, feedback
Data validation
Schema validation: column names, types, and constraints match expectations
Statistical validation: distributions have not shifted beyond thresholds
Completeness checks: no missing partitions, expected row counts
Freshness checks: data is not stale
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
Tests · Verify that the null check correctly identifies the null user_id row. Verify the range check catches the negative value. Add your own duplicate check.
Feature stores decouple feature computation from model training:
Tools like Feast, Tecton, and Hopsworks provide:
Consistent features between training and serving (avoiding training-serving skew)
Point-in-time correctness (no future data leaking into training features)
Feature reuse across multiple models and teams
What Do You Think?
A model trained on features computed offline performs well in evaluation but poorly in production. What is the most likely cause?
Training-serving skew is one of the most insidious bugs in ML systems. If your training pipeline computes a feature using a Pandas window function but your serving pipeline uses a SQL query, subtle differences (handling of nulls, edge cases, time zones) will cause the model to see different feature distributions at serving time than it saw during training. Feature stores exist specifically to solve this problem.
This is the stage most people think of when they hear "machine learning." But even here, production training is vastly different from notebook experimentation:
Reproducibility requirements
Pinned library versions (pip freeze, conda lock)
Fixed random seeds for data splits and initialization
Version-controlled training code, config, and data
Logged hyperparameters, metrics, and artifacts
Experiment tracking
Every training run records: hyperparameters, metrics over time, model artifacts, data version, code commit
Tools: MLflow, Weights & Biases, Neptune, Comet
Enables comparison across runs and reproduction of any result
Distributed training patterns
Data parallelism: replicate model across GPUs, split data
Model parallelism: split model across GPUs (for models too large for one GPU)
Pipeline parallelism: different layers on different GPUs, micro-batching
Your new model has 2% higher accuracy than the current production model on the test set. Should you deploy it?
A 2% overall accuracy improvement could mask a 15% degradation on your most important customer segment. Always evaluate on slices, not just aggregates. Production model validation is a multi-dimensional decision, not a single number.
Pick a statistic, set a threshold, alert when it breaks. PSI is the workhorse for tabular features; KS is the workhorse for continuous distributions. Run them yourself.
Here is the end-to-end ML pipeline as a continuous cycle. Each stage feeds into the next, and monitoring feeds back into data ingestion to trigger retraining:
Try it: Explore the End-to-End ML PipelineInteractive
Raw data arrives from databases, APIs, event streams, and data lakes. This stage handles batch pulls (Snowflake, BigQuery), streaming ingestion (Kafka, Kinesis), and external enrichment. Every batch goes through schema validation, completeness checks, and freshness verification before proceeding.
Raw data is transformed into informative features -- numerical scaling, categorical encoding, temporal aggregations, text embeddings. A feature store (Feast, Tecton) ensures identical feature computation between training and serving, preventing the dreaded training-serving skew.
Features are pulled from the store and fed to the model. Every run logs hyperparameters, metrics, code version, and data version to an experiment tracker (MLflow, W&B). Distributed training on GPU clusters with checkpointing handles large-scale jobs.
Trained model artifacts are versioned and stored in a registry. Each version is tagged with its training run, dataset version, and evaluation metrics. The registry enables promotion workflows: Development to Staging to Production, with automated gates at each transition.
Models move from the registry to serving infrastructure via shadow mode (run alongside production without serving users), canary deployment (route 1-5% of traffic), or blue-green switching. Each strategy trades speed against safety.
Track data drift (PSI, KS tests), prediction drift, model confidence, latency, throughput, and business metrics. Monitoring detects when the world changes and the model needs retraining. Alerts fire when distribution shifts exceed thresholds.
When monitoring detects significant drift or performance degradation, it triggers the pipeline to cycle back to data ingestion. The pipeline is not linear -- it is a continuous loop. Automated retraining keeps models fresh as data distributions evolve.
The ML code is the tip of the iceberg. Training is a tiny fraction of a production ML system; the majority of effort goes into data pipelines, feature stores, serving infrastructure, monitoring, and testing
Every pipeline stage has distinct failure modes. Data ingestion can fail silently (schema changes), training can diverge (hyperparameter issues), and serving can degrade (distribution drift); build defenses at each stage
Reproducibility requires versioning everything. Data, code, model weights, hyperparameters, and environment configurations must all be versioned together to reproduce any past result or debug any production issue
Automate pipelines from day one. Jupyter notebooks that run once are not pipelines; production systems need orchestrated, scheduled, idempotent workflows that run reliably without human intervention
You now have the full picture of what it takes to get ML from a notebook to production. Next up: MLOps -- the discipline of automating and managing this entire pipeline with CI/CD practices borrowed from software engineering.