A sepsis model scores 95% accuracy. It misses 8 out of 11 actual sepsis cases. If you only know accuracy, you can't tell a useful model from a deadly one. Here are the metrics that actually matter.
Learning Objectives
After this lesson, you will be able to:
Read a confusion matrix and calculate the key metrics from it: precision, recall, F1, and accuracy
Understand ROC curves (a visual way to compare models) and know when to use ROC-AUC vs. Precision-Recall AUC
Set up cross-validation (testing your model multiple times on different slices of data) and understand why a single random split can be misleading
Pick the right success metric for your specific problem — because the best metric for a spam filter is different from a cancer detector
Don't worry if all these metrics feel overwhelming — in practice, you usually only need two or three of them. This lesson gives you the full toolkit, and you will develop an instinct for which ones matter with experience!
#War Story: The 95%-Accurate Model That Was a Disaster
A healthtech startup celebrated when their sepsis-prediction model hit 95% accuracy on a hospital validation set. They demoed it to the board. They cleared the pilot. Three months in, the medical director called an emergency meeting: the model had missed 8 of 11 actual sepsis cases in the pilot ward.
The reveal:
Metric
Value
What it told us
Accuracy
95%
"Looks great!"
F1
0.41
"Wait, that's bad."
ROC-AUC
0.92
"Ranking is fine..."
Recall (sepsis class)
27%
"We're missing 73% of cases."
What happened? Sepsis was 4% of cases. A model that always predicted "no sepsis" would have hit 96% accuracy. The 95%-accurate model was barely better than predicting nothing. It had also been compared against a 50/50 threshold; with a 0.92 AUC, recalibrating the threshold could have pushed recall to 85% — at the cost of more false alarms that nurses could triage.
If you only know accuracy, you can't tell a useful model from a useless one. Worse: you can't tell whether the model is almost good and just needs a threshold change (AUC says yes) or fundamentally broken (AUC near 0.5). This lesson gives you the full diagnostic toolkit.
The confusion matrix is the foundation of all classification metrics. Drag the threshold slider and switch between class-imbalance presets — watch how the four cells (TP, FP, FN, TN) react as you move the cutoff and how that ripples into every downstream metric.
Loading visualization...
The cost-aware mode lets you assign different dollar penalties to false positives and false negatives, then shows where the expected cost is minimized — almost never at the default 0.5 threshold.
Try it: Adjust the threshold and watch precision and recall trade offInteractive
Loading visualization...
Try this: Adjust the classification threshold and watch how the confusion matrix changes. Lowering the threshold catches more positives (higher recall) but also more false positives (lower precision). Raising the threshold is more conservative -- fewer false positives but more missed positives.
Quick check
A spam filter scores 99% accuracy on a corpus where 99% of email is legitimate. What does this tell you about the model?
Try it! Open the Python REPL and type these lines yourself. Build a quick confusion matrix: from sklearn.metrics import confusion_matrix; y_true = [1,1,0,0,1,0,1,0]; y_pred = [1,0,0,1,1,0,1,0]; print(confusion_matrix(y_true, y_pred)) — read the 2x2 grid: top-left is correct negatives, bottom-right is correct positives!
Optimize precision when: False positives are costly. Email filtering (do not flag real emails as spam), criminal justice (do not convict innocent people).
A cancer screening model has 95% precision and 60% recall. What does this mean in practice?
Quick check
A fraud detector has F1 = 0.0. Which of the following is necessarily true?
#Same model, three different metrics, three different stories
Same fraud-detection model evaluated on the same test set, but reported under three different metric framings:
Framing
Number
Story it tells
Accuracy
95.1%
"The model is correct on most transactions."
F1 (fraud class)
0.41
"Of fraud predictions, half are wrong AND we miss half of real fraud."
ROC-AUC
0.92
"The model ranks transactions well — fraud scores are usually higher than legit scores. Threshold is fixable."
All three are mathematically correct on the same model. Only one of them is actionable. The accuracy number gets reported in the all-hands; the F1 gets reported in the post-mortem; the AUC tells engineering "the model has real signal, we just set the threshold wrong." Picking the metric isn't a math question — it's a communication choice, and the right metric depends on whom you're communicating with.
Interactive Lab
Drag the threshold slider and watch TPR/FPR move along the ROC curve, then compute the AUC live. The best way to internalize 'AUC is threshold-independent' is to see it stay constant while the threshold moves.
AUC = 0.5: Random classifier (no better than coin flip)
AUC < 0.5: Worse than random (flip your predictions)
AUC=P(score(x+)>score(x−))for random positive x+ and negative x−
Build an ROC curve from scratch and watch the AUC update in real time as you drag the threshold. The PR curve view next to it makes the imbalanced-data trap visible — ROC looks great while PR collapses.
Loading visualization...
Quick check
A binary classifier reports ROC-AUC = 0.5. What does this mean?
ROC-AUC can be misleading with highly imbalanced data. With 99.9% negatives, FPR stays low even with many false positives (because TN is huge). Use Precision-Recall AUC instead:
pythonrunnable cell
1
2
from sklearn.metrics import average_precision_score
pr_auc = average_precision_score(y_true, y_scores)
Rule of thumb: Balanced data --> ROC-AUC. Imbalanced data --> PR-AUC.
What Do You Think?
You ship a fraud detector with the threshold set high enough that precision = 0.95. Six months in, you are told to 'increase recall'. What necessarily happens to precision?
Let's compute every metric we just discussed by hand on a deliberately imbalanced dataset, then compare against sklearn — this is the calculation an ML engineer reaches for when accuracy is lying and they need to see what is actually happening.
A single train/test split can be misleading -- the results depend on which examples happen to be in the test set. Cross-validation gives a more robust estimate:
For each fold: use it as the test set, train on the other K-1 folds
Compute the metric on each test fold
Report the mean and standard deviation across folds
K = 5 or K = 10 is standard. K = n (leave-one-out) gives the least biased estimate but is computationally expensive.
Interactive Lab
Step through the folds and watch which slice serves as train vs. test. The animation makes the leakage trap (where preprocessing learned on the full dataset contaminates each fold) impossible to forget.
A single train/test split is noisy — the score depends on which rows landed in the test set. Here are five hypothetical scores from the same model under five different random splits:
Split #
Accuracy
1
0.872
2
0.851
3
0.901
4
0.838
5
0.879
Mean ± std
0.868 ± 0.022
If you report only split #3, you're claiming 90.1% accuracy. If you report only split #4, you're claiming 83.8% — same model. The 5-fold average (0.868 ± 0.022) is the honest number. Reporting CV mean ± std also gives you a sense of how much improvement you need to claim a real win — anything inside one standard deviation is noise, not signal.
For classification, always use stratified K-fold, which ensures each fold has the same class proportions as the full dataset. Without stratification, some folds might have no examples of the minority class.
pythonrunnable cell
1
2
3
4
5
from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_idx, test_idx in skf.split(X, y):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
For time series data, standard K-fold is wrong because it leaks future information into the past. Use TimeSeriesSplit, which always trains on past data and evaluates on future data:
pythonrunnable cell
1
2
3
4
5
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
# Fold 1: train=[0-99], test=[100-199]
# Fold 2: train=[0-199], test=[200-299]
# etc.
Split data into train/validation/test (or use nested cross-validation)
For each candidate model and hyperparameter set: evaluate on validation set
Select the best model and hyperparameters based on validation performance
Final evaluation on the test set (once, never touch the test set during tuning)
#Grid Search vs. Random Search vs. Bayesian Optimization
Method
How It Works
Best For
Grid Search
Try all combinations
Few hyperparameters (<4), small grid
Random Search
Sample random combinations
Many hyperparameters, large ranges
Bayesian (Optuna)
Build a model of the objective
Expensive models, many hyperparameters
Best model=argθ∈ΘminK1k=1∑KL(fθ(−k),Dk)
What Do You Think?
You tune hyperparameters using 5-fold cross-validation on the training set, achieving F1=0.92. You then evaluate on the held-out test set and get F1=0.85. Why the gap?
Tests · Verify that accuracy of 'always negative' is 99%. Verify F1 of 'always negative' is 0. Compute metrics for a perfect model (TP=100, FP=0, FN=0, TN=9900). Verify all metrics are 100%.
Accuracy is misleading for imbalanced data. A model predicting the majority class always achieves high accuracy but catches nothing useful; use precision, recall, F1, or AUC for imbalanced problems
Precision and recall trade off against each other. Lowering the classification threshold catches more positives (higher recall) but also more false positives (lower precision); the right balance depends on the cost of each error type
ROC-AUC measures ranking ability across all thresholds. It answers "how well does the model rank positives above negatives?" and is threshold-independent, making it ideal for comparing models
Cross-validation gives more reliable performance estimates. K-fold cross-validation uses all data for both training and validation, reducing the variance of performance estimates compared to a single train/test split
Match your metric to the business cost of errors. In medical diagnosis, false negatives (missed disease) are costly so optimize recall; in spam filtering, false positives (lost real email) are costly so optimize precision
#The Bridge Forward: Probability Quality and Explanations
Accuracy, precision, recall, and AUC all measure whether the model is right or wrong. None of them measure whether the model's confidence is honest. A model that says predict_proba = 0.9 should be right 90% of the time on those predictions — and most aren't. That's the next subject: Probability Calibration.
Once you have an accurate AND calibrated model, the last step before production is explaining individual predictions — turning "the model says yes" into "the model says yes because of these three features." That artifact is what makes a model defensible in front of a regulator, a customer, or a court.
Congratulations -- you have completed Track 3: Classical Machine Learning. You now command the full arsenal of classical algorithms: regression, classification, trees, ensembles, SVMs, clustering, dimensionality reduction, and rigorous evaluation. Next up: Track 4 takes you into the world of Deep Learning, where neural networks learn representations that no classical algorithm can match.