"Andrew Ng calls it the dirty secret of ML: a deployed model spends 5% of its lifetime training and 95% of it being fed by a pipeline. When a pipeline breaks at 3 AM, no one cares which transformer architecture you chose — they care whether your DAG retries cleanly, validates schemas, and refuses bad data. Production ML is plumbing, and plumbing is what separates a demo from a system that earns revenue every quarter."
Learning Objectives
After this lesson, you will be able to:
Describe the stages of an ETL/ELT pipeline — basically, how raw messy data gets cleaned up step-by-step — and when to use each approach
Identify where ML data comes from (APIs, databases, files, live streams) and the pros/cons of each source
Design a simple data pipeline that takes messy raw data, cleans it up, checks it for errors, and outputs something an ML model can actually use
Explain why pipeline tasks must be idempotent and atomic, and build validation gates that halt the pipeline when data quality drops below acceptable thresholds
Every machine learning project begins not with a model, but with data. And that data almost never arrives ready to use. It comes from databases, APIs, CSV files, log streams, web scrapes, and manual spreadsheets -- each with its own format, quirks, and quality issues. The data pipeline is the system that turns this chaos into structured, validated, ML-ready datasets.
A startup has messy data from 12 different SaaS tools. They want to build ML models but don't know yet which features they'll need. Should they use ETL or ELT?
ETL (Extract, Transform, Load): Transform data before storing it. Best when storage is expensive, schemas are well-defined, or you need real-time cleanliness.
ELT (Extract, Load, Transform): Store raw data first, transform later. Best when storage is cheap, requirements are evolving, or you want to preserve raw data for future reprocessing.
For ML projects, ELT is almost always better. You rarely know upfront which features will be useful. Storing raw data lets you engineer new features without re-extracting from sources.
Data arrives from an astonishing variety of sources -- REST APIs, relational databases, CSV/JSON files, event streams, and web scrapes. Each source has its own format, schema, and quality level. The pipeline must connect to all of them reliably.
At this stage, nothing is cleaned or validated. You are simply identifying where the data lives and how to access it.
Pull data out of each source system without changing it. Use full extraction for small datasets, incremental extraction (only changed records) for large ones, or Change Data Capture (CDC) for real-time streams. The key principle: extract faithfully, transform later.
This is where raw data becomes useful. Normalize schemas across sources, cast types, deduplicate records, filter irrelevant rows, join tables using keys, and compute derived features like ratios and rolling averages. Each transformation is a function in a composable chain.
Before storing data, run it through a quality gate. Validate column schemas, check value ranges, verify uniqueness constraints, confirm referential integrity, and compare statistical distributions against baselines. Any record that fails validation goes to a dead letter queue for investigation.
Store the validated data in a format optimized for ML consumption -- data lakes (Parquet in S3/GCS), feature stores (Feast, Tecton), data warehouses (BigQuery, Snowflake), or vector databases (Pinecone, Weaviate). The destination depends on how the data will be consumed.
Deployment is not the end. Monitor pipeline health with retry logic, idempotency guarantees, throughput metrics, and alerting. When validation fails, when throughput drops, or when a source goes silent, the monitoring layer fires alerts before bad data reaches your model.
Batch pipelines process data in chunks at scheduled intervals (hourly, daily). They are simpler, cheaper, and sufficient for most ML use cases. Tools: Airflow, dbt, Spark, Prefect.
Streaming pipelines process data continuously as it arrives. Required when ML predictions must reflect the latest data within seconds. Tools: Kafka, Flink, Spark Streaming.
Most ML systems use batch for training (you retrain daily/weekly on historical data) and streaming for inference (you need fresh features when serving predictions in real time).
Data pipelines follow Extract-Transform-Validate-Load stages. Every pipeline, from a 10-line script to an enterprise system, takes raw data through extraction, cleaning, transformation, validation, and storage
ELT beats ETL for ML projects. Storing raw data first and transforming on demand lets you re-engineer features without re-extracting from sources, which is critical when feature requirements change frequently
Pipeline tasks must be idempotent. Re-running a failed task should produce the same result without duplicates or corruption, because pipelines fail regularly due to network issues, API timeouts, and disk problems
Validation gates prevent garbage from reaching your model. Schema checks, range validation, uniqueness checks, and distribution monitoring at each stage catch data quality issues before they degrade model performance
80% of ML time is spent on data. A well-architected data pipeline is the highest-leverage investment in any ML project, far outweighing model architecture decisions
What is the main advantage of ELT over ETL for ML projects?
The modern data stack in 2026 is built on the lakehouse pattern (Iceberg / Delta on S3), transformed via dbt or Polars/DuckDB for fast in-process work, served to ML through a feature store (Feast, Tecton), and observed by tools like Monte Carlo or Soda -- but none of that matters if your pipeline is not deterministic, idempotent, and replayable.
Your data pipeline is the foundation everything else rests on. Next up: Exploratory Data Analysis -- how to look at your data before you model it, and why that step saves you from countless downstream disasters.