Ensemble Methods: Bagging, Random Forests & Stacking
Why do Kaggle's top 10 entries almost always combine 100 models instead of training one strong one? Because the wisdom of crowds beats the wisdom of one — but only when the crowd disagrees in the right way. This is the math behind that.
Learning Objectives
After this lesson, you will be able to:
Explain why averaging many imperfect models beats a single fancy one — and why the math only works when the individual models are decorrelated
Build a Random Forest by combining bootstrap sampling with random feature subsets at each split, and use Out-of-Bag error as a free validation estimate
Tell Bagging, Random Forests, Extra Trees, and Voting ensembles apart, and pick the right one for a given dataset size and noise profile
Stack 3+ base models with a meta-learner using out-of-fold predictions to avoid leakage — the technique that wins Kaggle competitions
Recognize when AdaBoost is the right historical bridge to gradient boosting, then hand off cleanly to the dedicated Gradient Boosting Deep Dive lesson
Don't worry if "ensemble" sounds fancy — the core idea is just averaging a bunch of so-so models, and most of this lesson is variations on that one trick. Once you see why decorrelation makes averaging powerful, the rest clicks fast.
Playground:Ensemble Methods → — train Bagging, Random Forest, and Boosting side-by-side on the same dataset and watch how decorrelation improves accuracy as you add trees.
#Worked Example: Three 60%-Accurate Classifiers Can Hit 80%+
Imagine three independent binary classifiers, each correct 60% of the time, voting by majority. Assuming independence, the ensemble is wrong only if at least 2 of 3 are wrong:
P(all 3 wrong) = 0.4³ = 0.064
P(exactly 2 wrong) = 3 × 0.4² × 0.6 = 0.288
P(majority wrong) = 0.064 + 0.288 = 0.352
So the ensemble is correct 64.8% of the time? Not impressive. But with 5 independent 60%-classifiers:
With 9 independent 60%-classifiers, ensemble accuracy is 73.3%. With 25, it crosses 84.6%. This is Condorcet's jury theorem — as long as each voter is better than random AND votes independently, the committee approaches 100% accuracy as M grows. The catch: real models are not independent, which is why the next sections engineer ρ (the correlation) downward.
The reason ensembles help is mathematical, not magical. Suppose you have M models, each with prediction variance σ². The variance of their average is:
Var(M1m=1∑Mfm(x))=M1σ2+MM−1ρσ2
This formula explains every design choice that follows. Bagging adds bootstrap diversity to lower ρ. Random Forests add feature-subset diversity to lower ρ further. Extra Trees add random splits to lower ρ even more. Stacking uses different model families (linear, tree, kernel) to push ρ as close to zero as possible.
Watch the same dataset get classified by progressively more decorrelated ensembles — single tree, then bagged trees, then a full Random Forest — and see how the decision boundary smooths as ρ drops.
EnsembleMethodsViz: single tree → bagging → random forest, side by sideInteractive
Loading visualization...
Quick check
Two practitioners both train 500-tree ensembles. Practitioner A's trees have average pairwise correlation ρ=0.05; Practitioner B's trees have ρ=0.8. Both single trees have variance σ²=1. By the variance formula Var(avg) = σ²/M + (M−1)/M · ρ · σ², roughly how much variance reduction does each get?
Create B bootstrap samples (random samples with replacement) from the training data — each is the same size as the original.
Train one base model (typically a deep, fully-grown tree) on each bootstrap sample.
To predict: average the B model outputs (regression) or majority-vote (classification).
f^bag(x)=B1b=1∑Bfb(x)
Try it! Open the Python REPL and type these lines yourself. Compare a single tree vs. a bagged ensemble: from sklearn.tree import DecisionTreeClassifier; from sklearn.ensemble import BaggingClassifier; from sklearn.datasets import load_breast_cancer; from sklearn.model_selection import cross_val_score; X, y = load_breast_cancer(return_X_y=True); print(f"Single tree: {cross_val_score(DecisionTreeClassifier(), X, y).mean():.0%}"); print(f"Bagged 100 trees: {cross_val_score(BaggingClassifier(DecisionTreeClassifier(), n_estimators=100), X, y).mean():.0%}") — bagging wins.
What Do You Think?
Bagging gives one decision tree about 92% accuracy and a 100-tree bagged ensemble about 95% accuracy. You crank n_estimators up to 10,000. What happens to test accuracy?
The plateau answer is correct. Once tree predictions saturate the variance-reduction formula above (the second term ρ·σ² dominates), more trees don't help. The fix isn't more trees — it's less correlated trees. That's what Random Forests do next.
#Part 2: Random Forests: Adding Feature Randomness
Random Forests add a second layer of randomness on top of bagging: at each split inside each tree, only a random subset of features is allowed.
At each split: choose best from m random features (of d total)Classification: m=⌊d⌋Regression: m=⌊d/3⌋
Why restrict features at every split? Without this restriction, if one feature is strongly predictive (e.g. salary for income classification), every tree picks it for the root split. Trees become near-identical → ρ ≈ 1 → averaging buys nothing. Forcing some splits to use worse features is the price of decorrelation, and that price is overwhelmingly worth paying.
The trade is counter-intuitive: you make individual trees less accurate in exchange for the ensemble being much more accurate. This is one of the deepest insights in ML.
When each tree trains on a bootstrap sample, it sees about 63.2% of unique training rows. The remaining ~37% are out-of-bag for that tree — and they make a free validation set.
OOB error is one of the great practical features of Random Forests. It's why a Random Forest is the perfect "first model on a new dataset" — one .fit() call gives you both a model and an unbiased error estimate.
Extra Trees (Extremely Randomized Trees, Geurts et al. 2006) push randomness one step further: at each split, instead of finding the best threshold for the chosen feature, it picks a random threshold. This makes individual trees worse but ρ even smaller. On noisy datasets Extra Trees often beat Random Forests; on clean datasets RF usually wins. Always worth trying both.
pythonrunnable cell
1
2
3
from sklearn.ensemble import ExtraTreesClassifier
et = ExtraTreesClassifier(n_estimators=500, max_features='sqrt', n_jobs=-1)
et.fit(X_train, y_train)
Compare Bagging, Random Forests, and Extra TreesInteractive
Loading visualization...
Quick check
Why does bagging — averaging many models trained on bootstrap samples — primarily reduce VARIANCE rather than bias?
#Part 3: Voting Classifiers: Different Algorithms, Same Vote
Bagging and Random Forests use one model family (trees) but vary the data. Voting classifiers flip this: use different model families on the same data and combine their votes.
Hard voting: each model votes for a class, majority wins.
Soft voting (almost always better): each model outputs class probabilities, average the probabilities, pick the class with the highest mean probability.
pythonrunnable cell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
vote = VotingClassifier(
estimators=[
('lr', LogisticRegression(max_iter=1000)),
('rf', RandomForestClassifier(n_estimators=500)),
('svm', SVC(probability=True)),
],
voting='soft'
)
vote.fit(X_train, y_train)
Voting is a poor cousin of stacking (next section) — it doesn't learn how to combine the base models, it just averages them. Use voting when you need a quick uplift; use stacking when you need every last bit of accuracy.
#Part 4: Stacking: Meta-Learners on Out-of-Fold Predictions
Stacking is the technique that wins Kaggle competitions. The setup:
Choose a set of base models that make different kinds of mistakes (a tree-based model, a linear model, a kernel-based model, sometimes a neural net).
Generate out-of-fold (OOF) predictions from each base model using k-fold CV on the training set. Critically, every prediction must come from a model that did not see that row during training.
Treat those OOF predictions as new features and train a meta-learner (typically logistic regression or a small neural net) on top.
At inference time: refit each base model on the full training set, get test predictions, feed them to the meta-learner.
Train: Zi,m=fm(−k(i))(xi)Meta: y^i=g(Zi;θ)where k(i) is the fold containing row i
The whole game is the OOF discipline. If you train each base model on the full training set and feed its training-set predictions to the meta-learner, the meta-learner just learns to copy the base models' overfitting and the whole thing collapses. Sklearn's StackingClassifier handles the OOF logic for you — but if you implement it yourself, this is the bug to watch for.
Why stacking wins: the base models make different kinds of mistakes. A Random Forest fails on smooth linear trends. A logistic regression fails on non-linear interactions. A KNN fails when the wrong distance metric is in play. The meta-learner learns which model to trust on which kind of input — a level of adaptivity that no single algorithm can match.
What Do You Think?
You stack four base models (RF, GBM, logistic regression, KNN) on a 10K-row dataset and the stacked model scores worse than the best single model. The most likely cause?
The OOF leakage answer is the canonical one. The meta-learner trained on training-set predictions sees suspiciously perfect base-model outputs and learns to over-trust them, which destroys generalization. Always use the cross-validated StackingClassifier (or the mlxtend equivalent) — never roll OOF logic by hand unless you really know what you're doing.
AdaBoost (Freund & Schapire 1995) is the historical foundation of boosting, and a natural bridge from this lesson to the next. The idea: train weak learners (typically decision stumps — depth-1 trees) sequentially. Each stump pays more attention to the examples the previous stump got wrong by upweighting them in the loss.
FM(x)=m=1∑Mαmhm(x)where αm=21lnϵm1−ϵm
In modern practice AdaBoost is rarely the best choice — gradient boosting almost always beats it on tabular data. But understanding AdaBoost makes the gradient-boosting lesson easier: gradient boosting generalizes the same idea (sequential error correction) to arbitrary differentiable loss functions.
Quick check
Boosting (sequential, each model targets previous errors) is said to reduce BIAS more than bagging. Why?
Forward pointer: Boosting is the other major ensemble family — sequential models, each fixing the previous one's mistakes. The next lesson, Gradient Boosting Deep Dive: XGBoost, LightGBM, CatBoost, takes this further with the algorithm that dominates tabular industry and Kaggle.
#Random Forest vs. Gradient Boosting: Quick Comparison
Aspect
Random Forest
Gradient Boosting
Training
Parallel (fast)
Sequential (slower)
Hyperparameter sensitivity
Low (defaults work well)
High (must tune learning_rate, n_estimators)
Bias
Higher (independent trees)
Lower (corrects errors)
Variance
Lower (averaging)
Tunable via depth + regularization
OOB error built in
Yes (free CV)
No (need explicit val set)
Best for
Quick baseline, robust results
Maximum accuracy when tuned
The next lesson covers gradient boosting in depth.
#Single Tree vs Random Forest vs Voting: Side by Side
Run all three on the breast-cancer dataset in the same cell. You'll see the single tree wobble around 90%, the Random Forest jump to ~96%, and a soft-voting ensemble of three diverse model families typically match or beat the forest.
Tests · Verify the stacked model's CV score equals or exceeds the best individual base model. Confirm test accuracy is in line with CV estimate. Try replacing one base model with a copy of another (e.g., two RFs with different seeds) and watch the stack gain shrink — diversity is what makes stacking work.
Averaging only works when models are decorrelated. The variance of an averaged ensemble is σ²/M + (M-1)/M·ρ·σ²; the entire engineering of bagging, RFs, Extra Trees, and stacking is a hunt for low ρ
Random Forests = bagging + random feature subsets at each split. This two-source randomness (data + features) decorrelates the trees enough that averaging buys real variance reduction; OOB error is a free cross-validation estimate
Voting is averaging across model families; stacking is learning across model families — voting just averages base-model probabilities; stacking trains a meta-learner on out-of-fold predictions and beats voting by 0.5-1.5% when base models are diverse
Out-of-fold discipline is the whole game in stacking. Every meta-feature must come from a base model that didn't see that row during training; sklearn's StackingClassifier handles this for you, but it's the #1 thing to verify if a hand-rolled stack underperforms
AdaBoost bridges to gradient boosting. Sequential error correction with reweighted examples; in modern practice gradient boosting (XGBoost / LightGBM / CatBoost) replaces AdaBoost on tabular data, and that's the next lesson
Why does Random Forest force each split to consider only sqrt(d) features?
You now know the parallel-ensemble family — bagging, Random Forests, voting, and stacking. The other half of the toolkit is the sequential family. Next up: Gradient Boosting Deep Dive — XGBoost, LightGBM, and CatBoost, the algorithms that dominate Kaggle and most tabular industry use cases.