In 1936, an English statistician named Sir Ronald Fisher was staring at four flower measurements and three iris species. He needed to draw a line — literally a line in feature space — that separated setosa from versicolor from virginica as cleanly as possible. The trick he invented that day, published as "The Use of Multiple Measurements in Taxonomic Problems," gave us two things at once: the Iris dataset (still the world's most-loved teaching dataset 90 years later) and Linear Discriminant Analysis — a method so foundational it now lives quietly inside every credit-score model, every early facial-recognition pipeline, and every "is this MRI tumor benign or malignant" classifier. The math is two scatter matrices, one generalized eigenvalue problem, and one decision rule. By the end of this lesson you'll know exactly when LDA crushes Gaussian Naive Bayes, when QDA crushes LDA, and when both of them get crushed by something else.
Learning Objectives
After this lesson, you will be able to:
Set up Fisher's criterion — maximize between-class scatter over within-class scatter — and derive LDA as a generalized eigenvalue problem
Recognize LDA as a CLASSIFIER (under shared Σ, linear boundary) and as a DIMENSIONALITY-REDUCTION method (project to top C-1 Fisher directions)
Place Gaussian NB, LDA, and QDA on a single flexibility spectrum: diagonal-Σ → shared full Σ → per-class full Σ
Diagnose when LDA wins (small data, homoscedastic, roughly Gaussian classes) vs. when QDA wins (lots of data per class, visibly different covariances) vs. when neither works (non-Gaussian, heavy-tailed, or outlier-ridden classes)
Distinguish LDA (supervised, finds class-separating directions) from PCA (unsupervised, finds max-variance directions) and explain why they generally point in different directions
Justify the C-1 dimensionality cap of LDA from the rank of the between-class scatter matrix
Credit scoring still uses LDA-style discriminants. The original FICO score, the Altman Z-score for corporate bankruptcy prediction, and countless modern credit-risk pipelines compute a linear discriminant of borrower features (income, debt ratio, payment history) and threshold it; LDA is interpretable, regulator-friendly, and outperforms heavy ML when sample size per class is small
Medical diagnosis with a handful of biomarkers. When a lab measures 5-20 continuous biomarkers per patient on a few hundred patients per class (benign vs. malignant), QDA's flexibility-vs-overfitting tradeoff often beats deep models and matches gradient boosting; the per-class covariance structure literally encodes "which biomarkers move together within healthy patients vs. within sick patients"
Build this --> classify iris flowers four ways — Gaussian NB, LDA, QDA, and PCA-projected-then-NB — on the very dataset Fisher created in 1936, and see when each variant wins on the data-generating process it was designed for
Don't worry if "generalized eigenvalue problem" sounds intimidating — the picture in your head can stay simple: each class is a fuzzy blob in feature space, and Fisher's question is "which direction makes the blob centers furthest apart relative to how fuzzy the blobs are?" That's it. The math just makes that question precise.
Gaussian Naive Bayes, LDA, and QDA are siblings. They all assume each class generates points from a multivariate Gaussian, and they all classify a new point by picking the class whose Gaussian most likely produced it. The only thing that changes is what shape of Gaussian each class is allowed to wear.
Drag the class clouds around and toggle between shared and per-class covariance to watch LDA's straight boundary bend into QDA's curve.
We have C classes. For each class k, let μ_k be its mean vector and N_k its sample count. Let μ be the overall mean. Fisher asks: find a projection direction w that maximizes the ratio of between-class scatter to within-class scatter when the data is projected onto w.
Within-class scatter measures how spread out points are inside each class, summed over all classes. If class k has lots of internal noise, S_W will be large in directions where class k spreads out.
SW=k=1∑Ci∈class k∑(xi−μk)(xi−μk)⊤
Between-class scatter measures how far each class mean sits from the overall mean. Big eigenvalues of S_B point in directions where the class means are most spread apart.
SB=k=1∑CNk(μk−μ)(μk−μ)⊤
Fisher's criterion says: project onto the direction w that maximizes the ratio of between-class scatter to within-class scatter, both measured along w.
J(w)=w⊤SWww⊤SBw
Setting the gradient of J(w) to zero (and using the quotient rule) gives a generalized eigenvalue problem: the optimal w is an eigenvector of S_W^ S_B.
SW−1SBw=λw
This is the core algorithm. Three steps, three matrices, one eigendecomposition.
#LDA as a Classifier: Where Linear Boundaries Come From
Fisher's original paper used the projection as a classifier: project a new point onto w, then threshold. But there's a more principled view that comes from Bayes' rule applied to multivariate Gaussians.
Assume each class is multivariate Gaussian with mean μ_k and a shared covariance Σ (the homoscedastic assumption). Bayes' rule gives the log-posterior:
logP(y=k∣x)∝logπk−21(x−μk)⊤Σ−1(x−μk)
Expand the quadratic term and DROP the parts that don't depend on k (because they're the same across classes — they cancel when picking argmax). The cross term -2 x^T Σ^-1 μ_k survives, the x^T Σ^-1 x cancels (it's k-independent), and we get the LDA decision rule:
δk(x)=x⊤Σ−1μk−21μk⊤Σ−1μk+logπk
The key word in that math block is linear in x. δ_k(x) is (constant) · x + (constant) — a linear function. Comparing δ_k(x) to δ_j(x) gives a linear equation (something) · x = (something else). That equation defines a hyperplane in feature space. The decision boundary between any two classes under LDA is a hyperplane.
This is exactly Fisher's projection direction in disguise: the LDA decision boundary between class 0 and class 1 is perpendicular to Σ^{-1}(μ_1 - μ_0), which is the whitened direction between class means — Fisher's optimal projection.
QDA keeps the Gaussian story but lets every class wear its own covariance Σ_k. The math is the same Bayes-rule expansion, but now the x^T Σ_k^{-1} x term does depend on k (because Σ_k changes with class) and survives.
The -0.5 log|Σ_k| term is the shape penalty — classes with bigger, more spread-out covariance ellipses get penalized for being "diffuse." This is the same factor that shows up in the Gaussian normalization constant: a wide Gaussian assigns less density per point than a tight Gaussian.
#Three Generative Classifiers, One Flexibility Spectrum
This is the punch line. Gaussian NB, LDA, and QDA are NOT three different algorithms — they are one algorithm (fit Gaussians, classify by Bayes' rule) with three different restrictions on the covariance matrices.
Method
Covariance assumption
# Σ parameters (p features, C classes)
Boundary shape
Gaussian Naive Bayes
Each class: diagonal Σ_k
C · p (just variances per class)
Quadratic (per-class) BUT axis-aligned
LDA
Shared full Σ across classes
p(p+1)/2 (one full matrix)
Linear
QDA
Per-class full Σ_k
C · p(p+1)/2 (C full matrices)
Quadratic
Notice: Gaussian NB has FEWER parameters than LDA when p > C, even though it allows per-class covariances. That's because the diagonal restriction is severe. LDA spends those parameters on capturing within-class feature correlation; Gaussian NB doesn't, and that's its weakness on continuous correlated features.
Loading visualization...
Run that cell and you'll see LDA carving the three iris species into nearly-disjoint clusters in 2-D, while PCA — which has no idea about the labels — produces a layout where versicolor and virginica overlap heavily. PCA found the directions of maximum overall spread; LDA found the directions of maximum class-discriminating spread.
What Do You Think?
You have two classes in 2-D. Class 0 has covariance Σ_0 = [[1, 0], [0, 1]] (a tight circle). Class 1 has covariance Σ_1 = [[10, 0], [0, 0.1]] (a long thin horizontal ellipse). The class means are at (0, 0) and (5, 0). Which classifier gets the lowest test error?
Quick check
LDA can project a dataset with C classes into at most how many dimensions?
#PCA vs. LDA: Why They Generally Point in Different Directions
This confuses learners coming from unsupervised dimensionality reduction. Both PCA and LDA find "best" projection directions, but they answer fundamentally different questions.
PCA (unsupervised) — given just X (no labels), find directions of maximum variance. PCA solves the eigenvalue problem Σ_data w = λ w where Σ_data is the data covariance. The top eigenvector is the direction of greatest overall spread, ignoring labels.
LDA (supervised) — given X and y, find directions of maximum class-discrimination. LDA solves S_W^{-1} S_B w = λ w. The top eigenvector is the direction where class means are most spread apart, normalized by within-class noise.
Consider this clean example: two classes both have a long horizontal axis (high variance along x) and a tight vertical axis (low variance along y), but their means differ vertically. PCA's top direction is horizontal — that's where the overall variance is. LDA's top direction is vertical — that's where class means differ relative to within-class spread. Same data, opposite answers.
For face recognition this distinction is dramatic. The Eigenfaces method (PCA on face images) finds directions of high pixel-intensity variance, which often correspond to lighting — the most variable thing in face datasets! Fisherfaces (LDA on face images) finds directions where identity-discriminating features cluster, ignoring lighting. Belhumeur 1997 showed Fisherfaces cuts error rates dramatically under varying illumination.
Classes are roughly Gaussian (no heavy tails, no severe skew)
Within-class covariance structure really is similar across classes (homoscedastic)
Sample size is small relative to feature count
You need interpretable linear coefficients (regulators, doctors, loan officers like these)
You want fast training — LDA is a closed-form solution, no iteration
QDA wins when
Classes have visibly different covariance shapes (different orientations, different elongations)
You have lots of data per class — typically at least 10 · p samples per class
The extra flexibility is "earnable" — your training set can support estimating C separate covariance matrices
Neither works well when
Classes are non-Gaussian — heavy-tailed, multi-modal, or skewed → try a tree-based method or a kernel SVM
Features have strong outliers — the sample covariance is broken by outliers, both LDA and QDA inherit that brittleness → robust scatter estimators (MCD, Minimum Covariance Determinant) can help, but it's often easier to switch models
You have very high-dim sparse features (text bag-of-words) → Naive Bayes or logistic regression with L1/L2 dominates; LDA's covariance matrix is p × p and explodes in storage and computation as p grows
Loading visualization...
What Do You Think?
You're building a face recognition system in 1997 (deep nets don't exist yet). You have 40 people, 10 photos each (400 total). Each photo is 100×100 = 10,000 pixels. The PCA-based 'Eigenfaces' method and the LDA-based 'Fisherfaces' method are your two choices. Which generally wins on unseen test photos with varying lighting?
Quick check
You have 50 features and 4 classes. How many free parameters does each method estimate (just for the covariance structure — ignoring means and priors)?
This is where the family closes the loop. Recall Gaussian NB assumes each class's covariance is diagonal — features are conditionally independent given the class. LDA drops the diagonal restriction but adds the shared-Σ restriction. QDA drops both restrictions but pays in parameter count.
When does each shine?
Gaussian NB wins when features are roughly uncorrelated within each class — high-dimensional sparse continuous data (rare in practice for continuous features, more common after PCA-whitening).
LDA wins when feature correlations within a class are real and similar across classes — most natural sensor data fits this (medical labs, financial ratios, anthropometric measurements).
QDA wins when those correlations differ across classes — and you have data to estimate them.
If you fit all three on the same data and rank them by test accuracy, the winner tells you something about the data-generating process:
LDA >> QDA → covariances really are shared; QDA's extra parameters added variance without bias reduction
QDA >> LDA → covariances really do differ across classes
Gaussian NB ≈ LDA → within-class feature correlations are weak (close to diagonal already)
Gaussian NB << LDA → within-class correlations are real and LDA is capturing them
This is why fitting all three is a standard diagnostic step on a new tabular classification problem — the relative performance is more informative than any single accuracy number.
Credit scoring (Altman Z-score, 1968). Edward Altman trained an LDA classifier on five financial ratios (working capital / total assets, retained earnings / total assets, EBIT / total assets, market value of equity / book value of debt, sales / total assets) to predict corporate bankruptcy within two years. The resulting Z = 1.2·X₁ + 1.4·X₂ + 3.3·X₃ + 0.6·X₄ + 1.0·X₅ is literally the LDA decision function written out. Banks still use Z-score variants as a regulatory benchmark in 2026 because LDA is auditable: every coefficient is a published number, regulators can challenge each one, and there's no "model card" debate.
Medical diagnosis. Imagine 200 patients with 10 biomarkers (cholesterol, white-cell count, blood pressure, etc.), split into "healthy" vs "early-stage disease." LDA with 10 features and 200 samples sits exactly in the sweet spot: enough samples to estimate the shared covariance, few enough features that QDA would overfit. Medical researchers routinely report LDA discriminant functions as the "linear combination of biomarkers predictive of disease X."
Facial recognition before deep nets. Fisherfaces (Belhumeur 1997) dominated face recognition benchmarks from 1997 to roughly 2012. The recipe: flatten each face image into a pixel vector, fit LDA treating each person as a class, and project new faces onto the top C-1 Fisher directions. Match by nearest neighbor in projected space. The illumination-invariance property came essentially for free, because lighting variance lives in within-class scatter (S_W) and LDA actively suppresses it.
The Use of Multiple Measurements in Taxonomic Problems
Ronald A. Fisher (1936)
The original LDA paper, introducing both the Iris dataset and Fisher's linear discriminant. One of the most-cited papers in 20th-century statistics. The eugenics-journal venue is a sad historical artifact; the math is independent of it and remains foundational.
Eigenfaces vs. Fisherfaces: Recognition Using Class-Specific Linear Projection
Peter N. Belhumeur, João P. Hespanha, David J. Kriegman (1997)
Showed LDA-based Fisherfaces dramatically outperform PCA-based Eigenfaces on face recognition under varying illumination. The canonical demonstration that supervised dimensionality reduction beats unsupervised when the labels carry information about the noise structure.
LDA = Fisher's ratio. Find the projection that maximizes between-class scatter over within-class scatter; solution is the top eigenvectors of S_W^ S_B
Two faces of LDA. Under shared-Σ Gaussian assumptions, LDA is BOTH a classifier (linear boundary, Bayes-optimal) and a dimensionality reduction technique (project to top C-1 Fisher directions)
QDA is LDA with per-class covariance. Drops the shared-Σ restriction, gains quadratic boundaries, pays in parameter count (C × more covariance parameters)
Gaussian NB ⊂ LDA ⊂ QDA in flexibility, and the right choice depends on which assumptions your data actually satisfies
C-1 dimensionality cap. LDA gives at most C-1 projection directions because rank(S_B) ≤ C-1; this is a hard rank constraint, not a tuning knob
LDA vs. PCA. PCA maximizes total variance (unsupervised, ignores labels); LDA maximizes class-discriminating variance (supervised). They generally point in different directions, dramatically so when the dominant variance in your data is not the dominant signal for classification
Next up: K-Nearest Neighbors — the laziest classifier of all, where "training" is literally just remembering the data, and prediction means asking the closest examples to vote.