"'Garbage in, garbage out' is the oldest ML cliché — but in production it's a law of physics. Anthropic's training run on Claude 3 spent more compute on data quality filtering than on the model itself. Andrew Ng's 2024 'Data-Centric AI' talk made it official: 80% of ML engineering effort, in any serious team, is data quality work. The model architecture you choose matters; the data you feed it decides whether you ship."
Learning Objectives
After this lesson, you will be able to:
Spot the four biggest data problems: missing values, outliers (weird extreme values), duplicates, and inconsistencies
Classify missingness as MCAR, MAR, or MNAR — because the type of missingness determines which imputation strategy is safe to use
Pick the right fix for each problem — fill in blanks, remove bad rows, cap extreme values, or deduplicate — based on why the problem happened
Build a step-by-step data cleaning process that keeps a record of every change so you can always explain or undo what you did, and measure quality with completeness, uniqueness, and validity scores
Don't worry if this feels tedious — cleaning data is not glamorous, but it is genuinely the highest-impact skill in all of ML. Once you get the hang of it, it becomes second nature!
Data quality problems are the most common reason ML models fail in production. Not because the model architecture was wrong, not because the hyperparameters were suboptimal, but because the training data was dirty. Studies consistently show that improving data quality yields larger accuracy gains than improving model complexity.
Missing data is the most common quality issue. Values can be missing for different reasons, and the reason dictates the solution.
Types of missingness
MCAR (Missing Completely at Random): The missingness has no relationship to any variable. Example: a sensor randomly drops readings due to network glitches. Safe to drop rows or impute.
MAR (Missing at Random): Missingness depends on observed variables but not the missing value itself. Example: younger respondents skip the "retirement savings" question. You can predict the missing values from other columns.
MNAR (Missing Not at Random): Missingness depends on the missing value itself. Example: high-income people refuse to report income. This is the most dangerous type -- no imputation method can fully correct it.
Investigate first: Is it a data entry error (age = 999), a measurement error (sensor malfunction), or a genuine extreme value (a billionaire in an income dataset)?
Remove: If it is clearly an error. Document why.
Cap/Winsorize: Replace extreme values with a percentile threshold (e.g., cap at 99th percentile). Preserves the observation but limits its influence.
Try it! Open the Python REPL and type these lines yourself. Create a small array with an outlier: import numpy as np; data = np.array([10, 12, 11, 13, 999]); print(np.mean(data), np.median(data)) — notice how the mean is wrecked by the outlier but the median barely moves!
Log transform: Compresses the scale, reducing the influence of large values. Works well for right-skewed data.
Keep as-is: If outliers are genuine and informative (fraud detection, rare disease diagnosis), removing them removes the signal.
What Do You Think?
You are building a credit card fraud detection model. Your dataset has 0.1% fraudulent transactions, and they all look like 'outliers' compared to normal transactions. Should you remove them?
The answer highlights a critical principle: the definition of 'outlier' depends on the task. In fraud detection, the rare extreme transactions are exactly what you want the model to learn. In salary prediction, a billionaire's income might be a genuine outlier that distorts the model for everyone else.
Z-score and IQR are the entry points. For multivariate outliers (Mahalanobis), density-based detection (LOF, DBSCAN), and tree-based isolation (Isolation Forest), see the dedicated Outlier Detection lesson next.
Duplicate records inflate the apparent size of your dataset and bias your model toward the duplicated observations.
Types of duplicates
Exact duplicates: Identical across all columns. Easy to detect with hashing or exact comparison.
Fuzzy duplicates: Nearly identical but with minor differences (typos, formatting). "John Smith" vs "john smith" vs "J. Smith". Require string similarity metrics (Levenshtein distance, Jaro-Winkler).
Semantic duplicates: Different representations of the same entity. "NYC" vs "New York City" vs "New York, NY".
Inconsistencies are conflicting or invalid values:
Format inconsistencies: Dates as "2024-01-15" vs "01/15/2024" vs "Jan 15, 2024"
Unit inconsistencies: Mixing meters and feet, Celsius and Fahrenheit
Encoding inconsistencies: "Male"/"Female" vs "M"/"F" vs 0/1
Logical inconsistencies: End date before start date, age negative, graduation year before birth year
Try it: Clean a messy dataset step by stepInteractive
This dataset has missing values (red), outliers (orange), duplicates (yellow), and inconsistent formats (purple). Click each cleaning action and watch the data quality score improve from 45% to 95%.
Raw Data
|
v
1. Profile: Compute statistics, null rates, cardinality, distributions
|
v
2. Deduplicate: Remove exact duplicates, flag fuzzy matches for review
|
v
3. Validate types: Cast columns to correct types, flag failures
|
v
4. Fix inconsistencies: Standardize formats, units, encodings
|
v
5. Handle outliers: Investigate, then cap/remove/keep per column
|
v
6. Handle missing: Choose strategy per column based on missingness type
|
v
7. Validate constraints: Range checks, referential integrity, business rules
|
v
Clean Data + Cleaning Report (what was changed and why)
Tests · Verify duplicates are removed, invalid ages are caught, salary outliers are capped, and missing values are imputed. Check that the final dataset has no nulls.
Wasted compute: Training on dirty data wastes GPU hours on learning noise.
Wrong predictions: Models learn the noise, not the signal, leading to worse accuracy.
Debugging time: When a model fails in production, dirty data is the last thing people check but the most common cause.
Trust erosion: Stakeholders who see wrong predictions lose trust in ML, making future projects harder to fund.
IBM estimated that bad data costs the US economy $3.1 trillion per year. For ML specifically, practitioners report spending 25-50% of project time on data cleaning -- time that could be invested in modeling if data quality were higher upstream.
The four horsemen of bad data are missing values, outliers, duplicates, and inconsistencies — Each requires a different strategy, and the root cause (not just the symptom) determines the correct fix
Missingness type dictates the imputation strategy. MCAR is safe to drop or impute simply, MAR can be predicted from other columns, and MNAR (where missingness depends on the missing value itself) is the most dangerous and hardest to correct
Outliers are not automatically bad. In fraud detection or anomaly detection, the outliers ARE the signal; always investigate whether an extreme value is an error, a measurement artifact, or genuine data before deciding to remove, cap, or keep it
Clean training data the same way as production data. Every cleaning step must be encapsulated in a reproducible function applied identically during training and inference, and imputation statistics must be computed on training data only to prevent leakage
Data quality improvement yields larger gains than model complexity. Improving data quality consistently produces bigger accuracy improvements than switching to a fancier model architecture
You discover that 30% of 'income' values are missing, and the missing rate is much higher for respondents under 25. What type of missingness is this?
Data observability platforms (Monte Carlo, Soda, Great Expectations, Pandera) turn the rules in this lesson into automated checks that fail loudly at ingest time, before the bad batch corrupts downstream state.
With clean data in hand, it is time to make it powerful. Next up: Feature Engineering -- the art and science of transforming raw data into the representations that make models sing.