"In March 2020, every ML model trained on pre-COVID data drifted within weeks. Credit scoring broke. Demand forecasting broke. Ad CTR predictions broke. The companies that survived had data-drift monitoring already wired up. The companies that didn't paid out billions in mispriced risk and unsold inventory. This lesson is your insurance policy — and like all insurance, you only appreciate it after the disaster you avoided."
Learning Objectives
After this lesson, you will be able to:
Tell the three kinds of drift apart — covariate shift (input distribution moves), label shift (class balance moves), and concept drift (the input-to-output relationship itself breaks) — and recognize which one explains a given production failure
Detect distribution shift with the right test for the right data type — Kolmogorov-Smirnov for continuous features, chi-squared for categorical, and Population Stability Index (PSI) with the standard 0.1 / 0.25 alert thresholds for the headline metric every credit and risk team watches
Quantify how different two distributions are using KL divergence, Jensen-Shannon distance, and Wasserstein distance — and know why JS is what you actually ship when KL would explode on zero bins
Wire up data validation as code with Pandera (schema-level), Great Expectations (expectation-level), and unit tests for data — so a bad upstream feed fails CI instead of silently poisoning your model in production
Design the monitoring stack a real team runs: a frozen reference distribution, a rolling current window, drift checks per feature, alert thresholds, and a runbook that says exactly when to retrain, retire, or investigate
How COVID broke every retail forecasting model overnight in March 2020 -- toilet paper demand 10x'd, restaurant traffic went to zero, e-commerce displaced grocery. Models trained on 5 years of "normal" data started predicting last-Tuesday's pattern into a world that no longer existed. Walmart, Amazon, and DoorDash all switched to short-window models within weeks -- because the long history was now misleading, not informative
Build this --> Pick any feature from the dataset you've used through this track. Save its histogram from week 1 as your reference. Re-sample after a week, compute KS statistic and PSI between the two windows, and decide -- with thresholds you set in advance -- whether you'd alert
Don't worry if "drift" sounds like a fuzzy word -- by the end of this lesson you will have three sharp definitions and three tests that turn it into a number you can alert on.
A deployed ML model is the only piece of software that gets worse the longer it runs without anyone touching the code. Your auth service, your database, your CRON job -- they all keep working as long as the world they were built for keeps existing. Models depend on a stronger guarantee: the world they were trained on has to keep being the world they're predicting in. When that guarantee breaks, performance silently degrades. Detecting that the guarantee has broken is what this lesson is about.
#Three Kinds of Drift -- and Why the Distinction Matters
Not all drift is the same. The fix depends on what's actually moving. There are exactly three things that can shift, and each one calls for a different response.
This is the most common drift in practice. Marketing campaigns bring in younger users, a new geography opens up, a sensor gets recalibrated. The mapping from input to output is still valid; you're just seeing inputs you didn't see much of in training. Fix: retrain on a recent window, or apply importance weighting.
Label shift is the drift you can detect before labels arrive -- using your model's own predictions as a proxy. Fix: rebalance class weights, recalibrate decision threshold, occasionally retrain.
This is what killed Zillow. The buyers, sellers, and houses still looked statistically similar to a year ago -- so input drift checks would have all been green -- but what each combination of features meant for next-month price had changed. Concept drift can only be detected by watching realized outcomes. Fix: retrain on fresh labels, or, if labels are slow, switch to a shorter rolling window.
What Do You Think?
Your fraud model's predicted-fraud rate doubles overnight -- it was flagging 1.2% of transactions yesterday, 2.4% today. Accuracy on the labeled review queue is unchanged. Which kind of drift is this most likely?
The labeled accuracy is unchanged, so the model is still right when it gets to ground truth -- that rules out concept drift. The output rate moved without the model breaking, which is what label shift looks like from the inside: the world genuinely has more fraud today, the model is correctly catching it, the alert is real. Note that covariate shift could also drive this if the new inputs happen to be ones the model rates as risky -- which is why in practice you watch input distributions and prediction rate, and treat a divergence between them as the strongest signal.
#How to Measure Distance Between Two Distributions
You have a reference window (training data, or yesterday) and a current window (production today). Drift detection is one question: are these the same distribution? Three families of tests answer it.
Slide the current distribution away from the frozen reference and watch the drift metrics climb past their alert thresholds.
Loading visualization...
#The Kolmogorov-Smirnov Test -- Continuous Features
D=xsup∣Fref(x)−Fcur(x)∣
KS is non-parametric -- it doesn't assume normal, it doesn't assume anything about shape. That makes it a sane default for any continuous feature whose distribution you don't fully understand.
#Population Stability Index -- the Industry Standard
PSI=i=1∑B(ai−ei)⋅ln(eiai)
The + eps matters -- if a bucket goes from non-empty to empty (or vice versa) the log ratio explodes to infinity. Adding a tiny constant smooths it and is standard practice.
#KL Divergence and Why You Actually Ship Jensen-Shannon
For categorical features, KL/JS on the category-frequency vector is the natural test. For continuous, bin first (same edges as PSI) and then compute on the histograms.
#Validation as Code: Catch Bad Data Before It Reaches the Model
Drift detection assumes the data is at least valid. A separate, earlier line of defense is schema and expectation validation -- contracts asserted on every batch before it enters the system.
Schema-level (Pandera, pydantic): column names, types, nullability, ranges. Fast, deterministic, runs in CI on a sample and in production on every batch.
Expectation-level (Great Expectations, Soda Core): richer rules -- "mean is between X and Y", "fewer than 1% nulls", "this column is unique within day". These are unit tests for data: if they fail, the pipeline halts before the bad batch corrupts downstream state.
pythonrunnable cell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Pandera schema -- runs on every batch
import pandera as pa
from pandera.typing import DataFrame, Series
class Transaction(pa.DataFrameModel):
user_id: Series[str] = pa.Field(str_matches=r"^u\d+$")
amount: Series[float] = pa.Field(ge=0, le=1_000_000)
country: Series[str] = pa.Field(isin=["US", "CA", "UK", "DE", "JP"])
timestamp: Series[pa.DateTime] = pa.Field()
class Config:
strict = True # extra columns -> reject
coerce = False # type mismatch -> reject (don't silently cast)
# In the pipeline:
def ingest(batch: DataFrame) -> DataFrame[Transaction]:
return Transaction.validate(batch) # raises on violation
The principle: fail loud at the boundary, not silently three layers in. Every dataset that flows between systems should have a contract, and every contract should be machine-checked, not human-remembered.
Tests · Verify ks_statistic returns ~0 for no_drift and grows monotonically across scenarios. Verify PSI crosses 0.1 for small drift and 0.25 for big drift on this seed.
Compare distributions: shift one parameter and watch how fast each test reactsInteractive
Loading visualization...
In production, drift detection is not a one-off computation -- it's a continuously running system. Most teams converge on the same shape:
Frozen reference. A snapshot taken at deployment (or last retrain), stored in S3 / GCS, immutable. This is "what the model was trained for."
Rolling current window. Last N days or last N requests, recomputed daily.
Per-feature drift checks. KS / PSI / JS for every feature, plus the same for predictions and (when labels arrive) for performance metrics.
Alert thresholds set in advance. PSI > 0.25 fires page; PSI > 0.1 opens a ticket. Performance drop > 3% AUC fires page.
Runbook.Page on drift must answer: is the data feed broken? Has the population legitimately changed? Has the relationship broken? The first is a pipeline bug, the second is a retrain, the third is a redesign.
Drift comes in three flavors with three different fixes -- covariate shift (inputs move; retrain), label shift (class mix moves; recalibrate), concept drift (the X-to-y relationship breaks; redesign). Conflating them leads to either frantic retraining or silent failure
PSI is the industry-standard headline metric -- frozen reference, rolling current, the 0.1 / 0.25 thresholds. KS handles continuous, chi-squared handles categorical, JS divergence is the symmetric bounded distance you ship when KL would explode on zero bins
Validation as code is your first line of defense -- Pandera for schema contracts, Great Expectations for richer expectations. A bad batch should fail at ingest, not three pipeline stages later when the model is already serving wrong predictions
Concept drift is the only kind that requires labels to detect -- covariate and label shift you can catch with input distributions and prediction rates alone; concept drift only shows up when realized outcomes start disagreeing with the model. Build label-arrival monitoring as carefully as you build the model
A drift system is alerts + runbook + thresholds set in advance -- pick thresholds before you see the data, not after. The hardest part of monitoring is not measuring drift; it's deciding what severity to page on, and committing to that decision so on-call trusts the alerts
A bank's credit-default model has had stable performance for 18 months. Last week, PSI on the 'income' feature jumped from 0.04 to 0.31 over a single weekly window. Predicted default rate is unchanged and labeled accuracy on the review queue is unchanged. Most likely cause?
Modern observability tools (Evidently AI, NannyML, WhyLabs, Soda, Monte Carlo) package these tests into batteries you can run on a schedule. Before you reach for them, you should be able to compute PSI on a feature by hand; the playground below lets you do exactly that.
Loading visualization...
That closes the data foundations track. You can now ingest, explore, clean, encode, transform, reduce, split, validate, and monitor real data -- the eighty percent of ML work that determines whether the remaining twenty percent succeeds. Next: track-03 Classical Machine Learning. Now that your data is ready, watch your first algorithm find patterns in it.