Your model says 'I'm 80% sure.' The actual frequency of being right at that confidence level is 45%. That gap is calibration — and it's the difference between a useful probability and a confident-sounding number.
Learning Objectives
After this lesson, you will be able to:
Explain why a classifier saying 'I'm 90% sure' often does NOT mean it is right 90% of the time, and identify which models (SVM, Random Forest, Naive Bayes) are systematically miscalibrated and why
Diagnose calibration with reliability diagrams, Brier score, and Expected Calibration Error (ECE) — and know which metric tells you what
Apply Platt scaling (sigmoid) and isotonic regression via sklearn's CalibratedClassifierCV to fix miscalibrated models, and pick the right method based on dataset size
Avoid the leakage trap of calibrating on training data, design held-out or cross-validated calibration workflows, and monitor calibration drift in production
Don't worry if this feels new — most ML courses skip calibration entirely, even though it is the difference between a research demo and a production-grade model.
A trading firm built a Random Forest to predict whether a corporate bond would default within 12 months. The model's predict_proba output was used to size position limits: more confidence → larger position. Their risk engine logged every prediction and the realized outcome.
After six months, the analyst team plotted what the model said vs. what actually happened:
Model said
Bonds in this bucket
Actual default rate
10%
412
8%
30%
285
24%
50%
198
41%
70%
154
49%
80%
91
45%
90%
47
52%
Look at the 80% row. The model said 80% sure. The actual frequency was 45%. The risk engine was using these probabilities to compute expected losses — and the expected losses were off by 78%. The firm had been quietly under-reserving for almost half a year.
A CalibratedClassifierCV(method='isotonic', cv=5) wrapper and 15 minutes of compute fixed it. The new model's 80% bucket actually defaulted 81% of the time. Same accuracy. Same AUC. Honest probabilities. The difference between research and production is exactly this: calibration.
Different models break calibration in different ways:
Naive Bayes is over-confident because the conditional independence assumption breaks: when features are correlated, the probabilities multiply too aggressively, pushing outputs toward 0 or 1.
SVMs do not produce probabilities at all — they output unbounded decision-function scores. Any "probability" you read off predict_proba is a sigmoid hack.
Random Forests average bagged tree votes. Because individual trees rarely vote 0.0 or 1.0 unanimously, RF predictions stay clustered in the middle (under-confident at the extremes, over-confident in the middle band).
Boosted trees (XGBoost, LightGBM) with default learning rates and many rounds become over-confident as boosting drives margin upward without regularizing probabilities.
Logistic regression trained with maximum likelihood on a well-specified problem is, by construction, calibrated. This is the "free lunch" of LR.
The visual workhorse. You bin predictions by predicted probability, then plot the mean predicted probability on the x-axis vs. the empirical positive rate on the y-axis. A perfectly calibrated model traces the y = x diagonal.
Drag the threshold and switch base classifiers — the reliability curve shows you the exact shape of the miscalibration, and you can apply Platt scaling or isotonic regression and watch the curve snap toward the diagonal in real time.
Loading visualization...
For each bin Bm:conf(Bm)=∣Bm∣1i∈Bm∑p^i,acc(Bm)=∣Bm∣1i∈Bm∑1[y^i=yi]
ECE measures the average gap between confidence and accuracy across bins, weighted by bin size. It isolates the calibration component from discrimination.
ECE=m=1∑MN∣Bm∣acc(Bm)−conf(Bm)
What Do You Think?
Your random forest achieves 92% accuracy and 0.94 ROC-AUC, but the Brier score is 0.18 (relatively high). What is most likely going on?
Quick check
A linear SVM trained on a binary classification task outputs values like `-1.7`, `+0.4`, `+2.3` from its decision function. Why are those numbers not probabilities?
Fit a logistic regression on top of the model's raw scores. Originally developed for SVMs but works on any classifier that outputs a continuous score f(x).
Use Platt scaling when calibration samples are scarce (<1000) or when miscalibration is a clean sigmoid distortion (typical of SVMs).
Use isotonic regression when you have ample data (≥1000 calibration samples) and the miscalibration is non-sigmoid (typical of Random Forests, gradient-boosted trees, and modern neural networks).
Skip calibration for properly trained logistic regression on a well-specified problem — it is already calibrated, and adding a second stage can only add variance.
Quick check
You're calibrating two models: (A) an SVM with 600 training rows of a clinical dataset, and (B) a 500-tree gradient-boosted classifier with 100k rows of e-commerce click data. Which calibration method should you choose for each?
Quick check
Expected Calibration Error (ECE) over 10 bins is 0.04. What does that number actually represent?
Tests · Verify that the calibrated Brier scores are lower than the uncalibrated baseline. Check that ECE drops for both Platt and isotonic methods. Confirm the reliability gap shrinks after calibration.
Reliability diagram: Compare uncalibrated vs calibrated classifiersInteractive
Each point is a prediction-bin. The y=x diagonal is perfect calibration. Curves above the diagonal mean the model is under-confident in that bin; curves below mean over-confident. Watch how Random Forests stay clustered in 0.3-0.7 and how Naive Bayes pushes hard to the extremes.
Loading visualization...
A calibrated curve hugs the diagonal. An over-confident curve lies below the diagonal (model says 0.9 but is right only 0.7 of the time). An under-confident curve lies above the diagonal (model says 0.6 but is right 0.85 of the time).
#Concrete before/after: Random Forest under isotonic calibration
Same RF, same test set. Reliability values at each bin before and after CalibratedClassifierCV(method='isotonic', cv=5):
Mean predicted prob
Empirical positive rate (raw RF)
Empirical positive rate (isotonic)
0.10
0.18
0.11
0.30
0.42
0.31
0.50
0.61
0.50
0.70
0.65
0.69
0.90
0.71
0.88
The raw RF is under-confident at the low end (says 10%, reality is 18%) and over-confident at the high end (says 90%, reality is 71%) — the classic RF middle-clustering bias. After isotonic calibration, every bin lines up within 2 percentage points of the diagonal. Brier score on this run: 0.182 → 0.121 (33% drop). Accuracy and AUC unchanged. That's the entire promise of post-hoc calibration in one table.
Let's reproduce that same before-vs-after experiment live. Train a Random Forest, compute Brier and ECE, then overlay the raw, Platt-scaled and isotonic calibration curves on a single reliability diagram.
For multi-class problems, sklearn's CalibratedClassifierCV calibrates each class one-vs-rest, then renormalizes. This is not perfect — it can produce probabilities that do not sum exactly to 1 before renormalization — but it works well in practice. For deep learning, temperature scaling (a single scalar parameter dividing logits before softmax) is the dominant modern technique because it preserves the argmax (so accuracy is unchanged) while fixing over-confidence.
Calibration is whether stated confidence matches empirical accuracy. When a calibrated model says 0.7, it should be right 70% of the time; this is independent from accuracy and must be measured separately
Most classifiers are NOT calibrated by default. Naive Bayes pushes to extremes, SVMs need a sigmoid hack, Random Forests cluster in the middle, boosted trees over-confident, deep nets increasingly over-confident as they scale; only well-trained logistic regression is calibrated for free
Brier score, ECE, and reliability diagrams are the diagnostic trio. Brier captures calibration plus discrimination, ECE isolates calibration alone, the reliability diagram shows you where the miscalibration lives across the probability range
Platt scaling for small data and sigmoid distortions, isotonic for large data and complex shapes — sklearn's CalibratedClassifierCV(method='sigmoid' or 'isotonic', cv=5) is the one-line fix for nearly every classifier
Calibration drifts in production. Monitor ECE alongside accuracy, recalibrate on schedule, and segment by subgroup to catch hidden miscalibration that global metrics mask
Calibration is one of the last three checks before a model ships:
Model Evaluation told you whether the model is right. Calibration tells you whether the model knows whether it's right.
Model Interpretation explains why a prediction was made — but only if the prediction itself is honest. SHAP on an uncalibrated model attributes a distorted probability, which is misleading.
Hyperparameter Tuning maximizes a metric (often AUC) that says nothing about calibration. Tune for discrimination, then calibrate. Don't expect tuning to fix calibration; it won't.
Together: discriminate well (evaluation), tune for the right metric (tuning), explain transparently (interpretation), and report honest probabilities (this lesson). That's the full production-ready stack.
Calibrated probabilities are the bridge between a research-grade classifier and a production-grade decision system. Next up: clustering — turning the same data into structure when you have no labels at all.