"In 2009, the Netflix Prize winners shipped a recommender that won on the leaderboard and then never made production — the time-aware split they trained on didn't match how Netflix would actually serve predictions. The lesson is permanent: how you split your data IS your evaluation. Get the split wrong and every metric downstream is a lie that compounds for months until someone notices in production."
Learning Objectives
After this lesson, you will be able to:
Understand why you need three separate piles of data (training, validation, test) and what each one is for — and why looking at test results to make model decisions contaminates the evaluation
Split your data correctly for different situations — random splits, stratified splits for class imbalance, grouped splits for entity data, and chronological splits for time series
Spot and prevent the five types of data leakage — target leakage, train-test contamination, temporal leakage, group leakage, and external data leakage
Apply sklearn Pipelines to make preprocessing leakage structurally impossible by ensuring scalers and encoders only fit on training data within each cross-validation fold
Don't worry if train/test/validation feels confusing at first — it is really just the idea of practicing on some problems and saving others for the real test. You already do this when you study!
The train/val/test split is not just a best practice -- it is the foundation of honest model evaluation. Get it wrong and nothing else matters: your accuracy numbers are fiction, your model comparisons are meaningless, and your production deployment is a time bomb.
Your model gets 99% accuracy on the test set. Is it good?
When a model achieves suspiciously high accuracy, the first question should always be: "Is there leakage?" Real-world ML problems rarely yield 99% accuracy. If your model seems too good to be true, it almost certainly is.
Used to tune hyperparameters and make modeling decisions. The model never trains on this data, but you use it to compare models, select features, and decide when to stop training. Think of it as a practice test -- it guides your studying, but it is not the real exam.
The final, unbiased evaluation. Used exactly once -- after all modeling decisions are final. This is the real exam. If you peek at test results and then change your model, the test set becomes a second validation set and you need a new test set.
Try it! Open the Python REPL and type these lines yourself. Split a list into train and test: data = list(range(100)); train, test = data[:80], data[80:]; print(f"Train: {len(train)} examples, Test: {len(test)} examples") — you just did your first train/test split!
Here is exactly how a proper train/validation/test split unfolds, from a single dataset to three non-overlapping subsets used at different stages of model development:
Try it: Drag the split boundary and watch set sizes changeInteractive
You begin with a single labeled dataset. Every row has features (inputs) and a target (output). Before doing anything else, you need to decide how this data will be divided. The goal: create three subsets that serve three distinct purposes.
Randomly shuffle the rows to break any ordering artifacts (alphabetical, chronological, or by data source). Without shuffling, the first 70% might come from one source and the last 30% from another -- creating a biased split. For time series data, skip this step and use chronological ordering instead.
The first and largest portion becomes the training set. The model sees these examples repeatedly during training, adjusting its parameters to minimize loss. This is the data the model learns from.
The next portion becomes the validation set. Used to tune hyperparameters, compare model architectures, and decide when to stop training. The model never trains on this data, but you use validation results to make modeling decisions.
The final portion is locked away as the test set. This data is used exactly once -- after all modeling decisions are finalized. It provides the only unbiased estimate of how your model will perform on truly unseen data.
Feed the training set to your model. The model iterates over these examples multiple times (epochs), adjusting weights to minimize the training loss. Monitor both training loss and validation loss during this process.
After each training iteration, evaluate on the validation set. Use validation performance to tune learning rate, regularization strength, number of layers, and other hyperparameters. Stop training when validation performance plateaus or degrades (early stopping).
Only after your model is finalized, run it on the test set. This single number is your honest performance estimate. If you change anything after seeing test results, the test set is contaminated and you need a new one.
Without a validation set, you use the test set to tune hyperparameters. Each time you check test performance and adjust, you implicitly leak test information into your modeling decisions. After dozens of iterations, the test set is no longer an unbiased estimate of real-world performance.
The validation set is your "expendable" evaluation data -- you can look at it as many times as you want without compromising the test set's integrity.
When to use: Classification tasks, especially with imbalanced classes. If your dataset is 95% class A and 5% class B, a random split might put all class B examples in the training set, leaving the test set with no class B examples to evaluate.
When multiple examples come from the same source (same patient, same user, same experiment), they must all go into the same split.
pythonrunnable cell
1
2
3
from sklearn.model_selection import GroupShuffleSplit
gss = GroupShuffleSplit(n_splits=1, test_size=0.2)
train_idx, test_idx = next(gss.split(X, y, groups=patient_ids))
When to use: Medical data (multiple images per patient), user behavior data (multiple sessions per user), time series with multiple entities. If patient A's data appears in both train and test, the model might learn patient-specific patterns rather than generalizable medical knowledge.
For temporal data, you cannot randomly split. The training set must contain only past data, and the test set must contain only future data. Otherwise, you are using the future to predict the past -- which is impossible in production.
Time -------->
[---- Train ----][-- Val --][-- Test --]
Past Recent Future
Walk-forward validation extends this by sliding the window:
Data leakage occurs when information from outside the training data is used to create the model. It makes models appear much better during evaluation than they actually are in production.
A feature that is a direct consequence of the target variable, not a cause. Example: predicting hospital readmission using "discharge_date" -- patients who were readmitted have shorter discharge-to-readmission gaps, but you would not know the readmission date at prediction time.
Another example: predicting whether a loan will default using "number_of_collection_calls." Collections happen AFTER default -- this feature does not exist at the time you need to make the lending decision.
Data from the same entity appearing in both train and test. Examples:
Multiple X-rays from the same patient split across train and test
Multiple transactions from the same credit card in both sets
Augmented copies of the same image in both sets
The model learns patient/user-specific patterns instead of generalizable ones. Performance on the test set is inflated because the model recognizes "familiar" entities.
How do you know if you have leakage? Look for these red flags:
Suspiciously high accuracy: If your model achieves 99%+ accuracy on a problem that domain experts say is hard, something is probably wrong.
A single feature dominates: If one feature has 10x the importance of all others combined, investigate whether it is a proxy for the target.
Training and test performance are nearly identical: Some gap is normal (model generalizes imperfectly). If there is zero gap, the model might be seeing test data during training.
Performance drops drastically in production: The classic leakage symptom. The model was evaluating on leaked information that does not exist at prediction time.
With 99% negative, 1% positive: random split might put zero positives in a small test set. Use stratified splitting to guarantee proportional representation.
If your model will be deployed in a new city, your test set should contain data from cities not seen during training. Spatial autocorrelation means nearby locations have similar data -- random splitting would leak spatial information.
#Time-Series Splitting: Never Shuffle Temporal Data
pythonrunnable cell
1
2
3
4
5
6
7
8
from sklearn.model_selection import TimeSeriesSplit
# Time series cross-validation: always respects temporal order
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
# Train always comes BEFORE test in time
What Do You Think?
Your model has 99% accuracy on test data but 60% on real users. What went wrong?
The most likely culprit is data leakage. The 99% accuracy was never real -- it was inflated by information that leaked from the test set into training. Once the model faces genuinely unseen data from real users, the illusion shatters. This is why the gap between test accuracy and production accuracy is the most important diagnostic signal in ML.
Three-way splits serve distinct purposes. Training data teaches the model, validation data tunes hyperparameters and guides decisions, and the test set provides a single final unbiased evaluation that must not be peeked at during development
Data leakage is the silent killer of ML models. It makes models appear excellent in evaluation but fail catastrophically in production; preprocessing leakage (scaling before splitting) is the most common and most insidious form
Match your split strategy to your data type. Use stratified splits for imbalanced classes, grouped splits when multiple examples come from the same entity, and chronological splits for time series to prevent temporal leakage
Preprocessing parameters must come from training data only. Fitting scalers, imputers, or encoders on the full dataset (including test data) leaks information and inflates performance estimates; always fit on train, then transform test
Suspiciously high accuracy signals leakage. If your model achieves 99% on a problem domain experts consider hard, investigate for target proxies, future features, or train-test contamination before celebrating
You now have the complete data foundations toolkit: pipelines to collect data, EDA to understand it, quality checks to clean it, feature engineering to empower it, augmentation to expand it, and proper splitting to evaluate honestly. Next up: Track 3 -- Classical Machine Learning, where you will put this data to work building your first predictive models.