"K-fold is the default. But time-series needs forward chaining. Medical data needs Group K-fold. Imbalanced classes need Stratified K-fold. Hyperparameter tuning needs nested CV. Picking the right cross-validation strategy is more important than picking the right model — because the wrong CV scheme makes the wrong model look right, and that's how billions get spent on broken systems."
Learning Objectives
After this lesson, you will be able to:
Run k-fold cross-validation correctly and explain the bias-variance tradeoff between small k (faster, higher bias) and large k (slower, higher variance) so you can pick a sensible value (usually k=5 or 10)
Pick the right CV strategy based on data structure — stratified k-fold for class imbalance, GroupKFold for entity data (patients, users, sessions), TimeSeriesSplit for temporal data — and never random-shuffle a time series
Use nested cross-validation to get an unbiased estimate of generalization when you are also tuning hyperparameters, instead of overfitting to a single validation set
Compute out-of-fold (OOF) predictions for stacking, apply the Nadeau–Bengio variance correction when comparing two models, and use the 1-SE rule to pick the simplest model within one standard error of the best score
Build this --> Take any classification dataset on Kaggle, run vanilla KFold, StratifiedKFold, GroupKFold (using a synthetic group column), and TimeSeriesSplit on it; watch the score variance change and pick the strategy whose CV score most closely matches a held-out test set
Don't worry if the variants feel overwhelming -- they are all just different rules for which rows go into which fold, and you only need to pick the one that matches how your data was generated.
A single train/test split gives you one number — say, 87% accuracy — and you have no idea whether that number is reliable. Maybe the test set happened to contain easy examples. Maybe it had three hard outliers that knocked your score down 4 points. Cross-validation reuses every data point as both training data and test data, just never at the same time.
#K-Fold: Everyone Gets a Turn at Being the Test Set
The choice of k is a bias-variance tradeoff for the CV estimator itself, not the model.
Small k (e.g. k=2 or 3): Each training set is much smaller than the full dataset, so each model is weaker than what you'd ship. The CV score is biased downward (pessimistic).
Large k (e.g. k=N, which is LOOCV): Each training set is nearly the full dataset, so each model is almost identical to the final one — low bias. But the k models are highly correlated with each other (they share N-2 of N points), which makes the variance of the CV estimate high.
k=5 or k=10 is the practitioner's sweet spot. It is the default in scikit-learn, and the empirical literature (Kohavi 1995, Hastie-Tibshirani-Friedman ESL §7.10) finds these values give the best bias-variance tradeoff on most datasets.
Var(CVk)≈kσ2(1+(k−1)ρ)
Try it: Watch how data flows through the foldsInteractive
If you have a classification problem, KFold is almost never what you want — use StratifiedKFold instead. Stratified k-fold preserves the class distribution in every fold. If your data is 90% class A and 10% class B, every fold will be ~90/10 too.
What Do You Think?
You have a binary classification dataset with 950 negatives and 50 positives. You run plain KFold(n_splits=5, shuffle=True). What is the most likely failure mode?
With 50 positives split randomly across 5 folds, you might get folds with 6, 8, 10, 12, 14 positives. Per-fold metrics like F1 and AUC become extremely noisy at those small counts. Stratified k-fold deals one positive at a time, so each fold ends up with exactly 10 positives — your CV variance drops by an order of magnitude with no other change.
pythonrunnable cell
1
2
3
4
5
from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Pass y to .split() — stratification needs to see labels
for train_idx, test_idx in skf.split(X, y):
...
For multi-label or multi-target problems, see MultilabelStratifiedKFold from the iterative-stratification package.
#Group K-Fold: No Contestant Judges Their Own Performance
This is the most under-used CV strategy in industry, and the cause of the most embarrassing "the CV score was great but production tanked" stories. Use GroupKFold whenever your rows are not independent — patients with multiple visits, users with multiple sessions, sentences from the same document, photos from the same camera, transactions from the same merchant.
GroupKFold:∀i=j:groups(Di)∩groups(Dj)=∅
For classification with grouped data and class imbalance, scikit-learn 1.0+ added StratifiedGroupKFold, which respects both constraints at once. Use it whenever you have grouped imbalanced classification — it is strictly better than either alone.
What Do You Think?
Your dataset has 1,000 rows from 50 patients (each patient contributed ~20 visits), and you are predicting whether a visit results in re-admission. Class balance is 60/40. What CV strategy do you use?
StratifiedGroupKFold is the right answer — patients are the grouping unit (you must not train on Alice and test on Alice), and class balance is non-trivial enough that you want stratification too. Plain GroupKFold would also work but might give you one fold that is 75/25 by random luck.
For time series data, the only valid CV strategies are walk-forward designs that respect the arrow of time:
Expanding window (sklearn's TimeSeriesSplit): train set grows each fold; test fold is always the next chunk in time.
Sliding window: train set is a fixed-width window that slides forward; test fold is again the next chunk. Use when you suspect the relationship between features and target drifts (concept drift) and only recent data is informative.
Purged + Embargoed CV (López de Prado, finance): leave a small gap between train and test fold to handle features computed from rolling windows that span the boundary. Critical for high-frequency trading; useful for any dataset with engineered lag features.
pythonrunnable cell
1
2
3
4
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5, test_size=30, gap=7) # 7-day embargo
for train_idx, test_idx in tscv.split(X): # X must be sorted by time
...
The expanding-window picture: fold 1 trains on months 1-2, tests on month 3. Fold 2 trains on months 1-3, tests on month 4. And so on.
Leave-One-Out Cross-Validation (LOOCV) sets k=N. You fit N models, each leaving exactly one observation out. It is nearly unbiased because each model is trained on N-1 out of N data points (almost the full dataset).
LOOCV=N1i=1∑NL(f^(−i)(xi),yi)
When to use LOOCV:
Tiny datasets (N < 200) where you cannot afford to lose 20% of your data per fold to a 5-fold split.
Linear models with closed-form LOOCV (the PRESS statistic for OLS, the hat matrix shortcut for ridge regression) where you can compute it analytically without refitting N times.
When NOT to use LOOCV:
Large datasets. The compute cost is brutal and the variance is high anyway.
Classification metrics like AUC. Single-point test sets cannot compute meaningful classification metrics; the per-fold AUC is undefined for a single observation.
Repeated K-Fold runs k-fold multiple times with different random seeds and averages all of them. It reduces the variance of the CV estimate at linear cost. Use RepeatedKFold(n_splits=5, n_repeats=10) when you are comparing two models and need higher-confidence estimates than a single 5-fold can give.
Nested CV is the answer to a question you didn't know you needed to ask: if I tune hyperparameters using k-fold CV and report the best-fold score, is that score trustworthy? The answer is no — you have implicitly used the validation folds for model selection, so the reported score is biased upward by however much you searched.
pythonrunnable cell
1
2
3
4
5
6
7
8
9
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
inner_cv = KFold(5, shuffle=True, random_state=1)
outer_cv = KFold(5, shuffle=True, random_state=2)
grid = GridSearchCV(estimator, param_grid, cv=inner_cv)
nested_scores = cross_val_score(grid, X, y, cv=outer_cv)
# nested_scores is your unbiased estimate; grid.best_params_ from a final
# refit on all data is what you ship.
The cost is k_outer × k_inner model fits. For 5×5 nested CV with grid search over 20 hyperparameter combinations, that's 5×5×20 = 500 fits. Use RandomizedSearchCV with a small n_iter instead of full grid search if compute is tight.
When you run k-fold CV, every observation eventually appears in exactly one test fold and gets a prediction from the model trained on the other k-1 folds. Concatenating all those predictions gives you an out-of-fold (OOF) prediction vector — one prediction per training row, none of them computed by a model that saw that row in training.
OOF predictions are the foundation of model stacking: train base models, generate OOF predictions, then train a meta-model on the OOF predictions as features. Because OOF predictions are leakage-free, the meta-model learns to combine the base models without overfitting.
pythonrunnable cell
1
2
3
4
5
from sklearn.model_selection import cross_val_predict
oof_preds = cross_val_predict(base_model, X, y, cv=5, method='predict_proba')
# Now train a meta-model on oof_preds (leakage-free) + maybe original features
meta_model.fit(oof_preds, y)
When you compare model A against model B using paired k-fold CV scores, the naive standard error massively understates the true variance because the train sets across folds overlap heavily. Nadeau and Bengio (2003) derived the corrected variance:
VarNB(dˉ)=(k1+ntrainntest)⋅s2
For a final-model selection rule, use the 1-standard-error rule (Hastie-Tibshirani-Friedman ESL §7.10): pick the simplest model whose CV score is within one standard error of the best score. This biases toward simpler models and avoids overfitting to noise in the CV estimate itself.
Tests · Verify GroupKFold scores are lower than vanilla KFold (group leakage was inflating the latter). Verify TimeSeriesSplit's first fold trains on less data than the last fold.
A single train/test split is one lucky draw; CV averages many draws. K=5 or k=10 is the practitioner's default because it balances the bias of small-k (training sets too small) against the variance of large-k (highly correlated folds)
The CV strategy must match how the data was generated. Stratified for class imbalance, GroupKFold for entity-level data (patients, users, sessions, documents), TimeSeriesSplit for temporal data, StratifiedGroupKFold when both class imbalance and groups apply
Never random-shuffle a time series. Random KFold on temporal data lets the model train on the future and test on the past, inflating CV scores by 5-30 points relative to what you'll actually see in production
Tuning hyperparameters with k-fold and reporting the best score is silently biased. Nested CV (outer loop for evaluation, inner loop for tuning) is the only way to get an unbiased estimate of a tuned-pipeline's generalization
Compare two models with the Nadeau–Bengio correction, not the naive paired t-test. Paired k-fold scores share most of their training data, and the uncorrected variance underestimates the true uncertainty by a factor of 2-5×
You have a binary classification dataset with 950 negatives and 50 positives, and the rows are independent (no groups, not temporal). Which CV strategy is the best default?
When in doubt, ask one question: at deployment time, what is the unit of generalization (row, entity, time window)? Your CV iterator must mirror that boundary -- anything else is a number that disagrees with reality.
A trustworthy CV setup is the difference between weeks of confident iteration and weeks of chasing your own tail. Next up: Data Drift & Validation — what to do when the world changes after you train, and how to catch the drift before it embarrasses your model in production.