Spotify's recommender lives in a 200-dimensional taste space — but the playlist UI is a 2D grid. Netflix encodes every movie as a 4096-D feature vector, then squashes it into 50 dims before it ever hits the ranker. Your face on Apple Photos is a 512-D embedding compressed to 128 for on-device search. The common move? Take a wide cloud of points in high-dimensional space and find a much narrower view that keeps the structure you care about. That's dimensionality reduction, and it's how every modern ML system avoids drowning in its own features.
Learning Objectives
After this lesson, you will be able to:
Explain why dimensionality reduction matters — fighting the curse of dimensionality, enabling 2D/3D visualization, compressing for downstream models, and stripping noise out of features
Walk through PCA end-to-end: center the data, build the covariance matrix, eigendecompose it, project onto the top-k eigenvectors, and read off the explained variance ratio
Connect PCA to SVD — they are literally the same factorization, and SVD is the numerically stable way every library actually implements PCA under the hood
Know when to use LDA instead of PCA — when you have labels and Gaussian-ish classes, LDA picks directions that separate classes, not directions of max variance
Use t-SNE and UMAP responsibly for visualization — and recognize their famous traps (cluster sizes lie, between-cluster distances lie, perplexity matters, every run looks different)
Pick the right algorithm: PCA for linear + interpretable, LDA for supervised, t-SNE/UMAP for visualization only, autoencoders for non-linear compression you can invert
Don't worry if "high-dimensional space" sounds mystical — you already think in low dimensions all the time. When you describe a coffee in 2 numbers ("strong, bitter") instead of listing every chemical compound in it, that's dimensionality reduction. When you give directions as "two blocks north, one east" instead of listing your GPS coordinates, that's dimensionality reduction. The math just makes it rigorous.
There are four reasons you reach for dimensionality reduction in practice, and they're worth keeping straight because they push you toward different tools:
Visualization. You cannot plot a 784-D vector. You can plot 2-D. If the goal is a scatter plot for a paper, slide, or sanity-check, you want t-SNE or UMAP. PCA in 2D is often too faithful — it shows you the truth, which is that high-D data is mostly featureless blob.
Compression and speed. Every downstream model — KNN, SVM, logistic regression, even neural nets — gets faster and often better on fewer features. This is PCA's home turf: keep 95% of variance with 10% of the original dimensions and call it a day.
Noise reduction. The first few principal components carry the signal; the last few carry the dust. Truncating the small components is a denoising step. This is why image and audio compression (JPEG, MP3) are basically domain-specific PCA.
Beating the curse of dimensionality. In high-D, every point is far from every other point and distance metrics break. Drop to a manageable number of dimensions and your KNN, K-Means, and distance-based methods come back to life.
What Do You Think?
A standard MNIST digit is 28 × 28 = 784 pixels. How many principal components do you typically need to keep 90% of the variance?
Run PCA on a 2D Gaussian blob and watch the algorithm pick out the directions of maximum variance — those are your principal components.
Try it: Principal Component AnalysisInteractive
Drag the data cloud, change its spread, and watch the principal components rotate to track the new direction of maximum variance.
Loading visualization...
Try this: Stretch the cloud horizontally and watch PC1 align with the long axis — that's the direction of maximum variance. Now rotate the data 45 degrees and confirm the PCs rotate with it. The key insight: PCA does not care about the coordinate system you measured your features in. It finds the natural axes of the data itself.
PCA boils down to four steps. Each one is a couple of lines of numpy.
Step 1 — center the data. Subtract the column mean from every column so the cloud sits at the origin. PCA is sensitive to where the origin is; centering removes that arbitrariness.
X~=X−xˉ1⊤
Step 2 — build the covariance matrix. The d-by-d matrix that says, for every pair of features, how much they move together.
Σ=n−11X~⊤X~
Step 3 — eigendecompose the covariance. This is where the magic happens.
Σvi=λivi
To make the eigendecomposition concrete, watch how a matrix acts on its eigenvectors versus arbitrary vectors. Pin this intuition before you trust PCA on real data.
Try it: Eigenvector geometry — the foundation of PCAInteractive
Loading visualization...
Try this: Set the matrix to a sample covariance-like shape (large diagonal, small off-diagonal). The eigenvectors will be the principal axes; the eigenvalues will be the variances along them. PCA is exactly this picture, applied to the covariance of your data.
The eigenvalues are the variances along each PC. So the fraction of variance you keep with the top k components is simply:
EVR(k)=∑i=1dλi∑i=1kλi
This is your knob. Want maximum compression? Pick a small k and accept reconstruction error. Want lossless? Pick k = d and you've just done a rotation, not a reduction.
#PCA from Scratch + sklearn: and a Reconstruction Sanity Check
Let's run all four steps end-to-end on a real dataset, verify the numpy version matches sklearn, and watch the reconstruction error fall off as we keep more components.
Loading visualization...
Read the output: the three methods agree to 4+ decimal places (PCA-by-eigendecomp, PCA-by-SVD, and sklearn.decomposition.PCA are literally the same calculation). And reconstruction error drops exponentially at first — that's the steep part of the EVR curve — then flattens. The classical recipe is to read the elbow off this plot and call that your k.
Quick check
You run PCA on 1,000 images, each 256 × 256 pixels = 65,536 features. The first 10 components explain 92% of the variance. What does this tell you about your data?
Standard PCA finds a linear subspace. If your data wraps around a circle, a Swiss roll, or any curved manifold, the best linear subspace is a poor fit. Kernel PCA runs PCA in an implicit non-linear feature space defined by a kernel — most commonly an RBF (Gaussian) kernel. The result is principal components that can curve through the input space.
Kernel PCA is conceptually beautiful but practically tricky: choosing the kernel and its bandwidth is its own hyperparameter search, and the computation is O(n²) memory because you build the full Gram matrix. For most non-linear DR jobs you'll reach for UMAP or t-SNE instead.
#Incremental PCA for Streaming and Out-of-Memory Data
Standard PCA needs the whole dataset in memory to compute the covariance or SVD. When your data is 500GB or arrives as a stream, use Incremental PCA:
pythonrunnable cell
1
2
3
4
5
from sklearn.decomposition import IncrementalPCA
ipca = IncrementalPCA(n_components=50, batch_size=1000)
for batch in stream_batches(...):
ipca.partial_fit(batch)
X_reduced = ipca.transform(some_data)
Incremental PCA processes one batch at a time, updating its estimate of the principal components without ever materializing the full covariance. The trade-off is a small accuracy loss compared to full PCA, which is almost always invisible in practice.
PCA is unsupervised — it doesn't know about your labels. If you do have labels and your goal is to find directions that separate classes (not just directions of max variance), use Linear Discriminant Analysis.
The Fisher criterion that LDA optimizes:
J(w)=w⊤SWww⊤SBw
When LDA beats PCA:
You have labels (LDA is supervised; PCA is not).
Your classes are roughly Gaussian with similar covariance — LDA assumes this and will mislead you if your classes have wildly different shapes.
The direction of maximum between-class separation is different from the direction of maximum total variance. This is common when within-class variance is the dominant source of total variance.
When PCA wins anyway:
You don't have labels.
You only want compression / visualization, not class separation.
Your classes are non-Gaussian or have very different covariances (LDA's modeling assumption is violated).
LDA caps at C - 1 components for C classes: with 10 MNIST digits, you get at most 9 LDA components — fewer than you'd want for any compression task but plenty for a visualization that shows class structure.
Quick check
You have a sentiment classifier dataset: 50,000 movie reviews, each represented as a 5,000-dim TF-IDF vector, with binary positive/negative labels. You want to reduce to 50 dimensions before training a logistic regression. Should you use PCA or LDA?
#t-SNE: Non-Linear Visualization (With Big Asterisks)
For a 2D scatter plot that shows local neighborhood structure — points that were close in 50-D end up close in 2D — the modern default is t-SNE (t-Distributed Stochastic Neighbor Embedding, van der Maaten & Hinton, 2008).
Run t-SNE on the digits dataset and watch the 64-D pixel vectors collapse into well-separated 2D clusters that mostly correspond to the actual digit classes.
Try it: t-SNE on the digits datasetInteractive
Tune perplexity, learning rate, and iterations. Notice how the 'shape' of every cluster changes — but the cluster identities are stable.
Loading visualization...
Try this: Set perplexity to 5 and re-run. The clusters fragment. Set it to 50 and re-run — clusters merge into bigger blobs. The "right" perplexity is usually 5–50 depending on dataset size, but the takeaway is that t-SNE's appearance is very sensitive to a hyperparameter that has no principled default. This is the cost of its expressive power.
t-SNE has two halves. First, in the original high-D space, it converts pairwise distances into conditional probabilities — "what's the chance point i picks point j as a neighbor?" — using a Gaussian centered on i whose width is set by the perplexity. Then, in the low-D (usually 2D) embedding, it does the same trick but with a heavy-tailed Student-t distribution instead of a Gaussian. Finally it adjusts the 2D points to minimize the KL divergence between the two distributions of neighbor probabilities.
KL(P∥Q)=i=j∑pijlogqijpij
Perplexity (typical values 5–50) is the most important hyperparameter: it's an effective number of neighbors each point considers. Low perplexity = local structure dominates. High perplexity = global structure dominates. But — and this is the famous part — different perplexities give qualitatively different plots, and there's no single right answer.
Two clusters in a t-SNE plot are sitting very close together — almost touching. What can you reliably conclude about the points in those clusters in the original high-dimensional space?
Vanilla t-SNE is O(n²) in both time and memory — fine for 10,000 points, painful for 100,000, infeasible above. Modern implementations (Barnes-Hut t-SNE, FFT-based FIt-SNE, openTSNE) bring it down to O(n log n), but t-SNE still tops out around a few hundred thousand points. For millions, switch to UMAP.
UMAP (Uniform Manifold Approximation and Projection, McInnes et al., 2018) is the modern alternative to t-SNE. The math is rooted in algebraic topology — UMAP builds a fuzzy simplicial complex on the high-D data and finds a low-D layout whose fuzzy complex is as close as possible — but the practical behavior is "t-SNE with two important upgrades":
Faster. UMAP scales to millions of points where t-SNE chokes around 100K. Most of the speed comes from approximate nearest neighbors and a stochastic gradient descent–style optimizer.
Better global structure. UMAP usually preserves the relative positions of clusters better than t-SNE, so the between-cluster geometry of a UMAP plot is more (though still not perfectly) trustworthy.
n_neighbors (default 15): like t-SNE's perplexity. Low = local structure, high = global structure.
min_dist (default 0.1): how tightly points are allowed to pack in the embedding. Low values produce tight clusters; high values give a more spread-out, blobby look.
UMAP has an honest .transform() method, so you can fit on a training set and project new points later — something vanilla t-SNE genuinely cannot do. This makes UMAP a viable component in production pipelines (think: visualizing live embeddings in a dashboard as new data arrives) where t-SNE just isn't.
Run both on the same dataset, time them, and compare the cluster structure. This is the playground where most people actually decide which one to reach for in their day job.
Loading visualization...
Sample output: PCA runs in milliseconds and produces an overlapping mess that the human eye cannot cluster. t-SNE takes 10–30 seconds and gives a clean 10-cluster plot that obviously corresponds to the 10 digit classes. Isomap (the UMAP cousin) is in between on both axes.
Both PCA and LDA are linear DR methods. Both t-SNE and UMAP are non-linear but produce no general-purpose mapping from input to embedding (t-SNE has no transform(); UMAP's is approximate). The deep-learning answer is the autoencoder: a neural network with a bottleneck.
You train it to reconstruct its own input, minimizing MSE between input and output. The 8-D activations at the bottleneck are your non-linear, learned embedding. Unlike t-SNE/UMAP, autoencoders give you:
A deterministic forward pass — same input, same embedding, every time.
A real transform() for new data (just run the encoder).
An invertible mapping back to the input space via the decoder.
When you sub in a probabilistic bottleneck you get a VAE (variational autoencoder); when you swap MSE reconstruction for an adversarial loss you get the autoencoder backbone of GANs and diffusion. We pick all of this up properly in the Deep Learning track — for now, just know that autoencoders are the non-linear, parameterized cousin of PCA, and that PCA is exactly what an autoencoder with a single linear layer and MSE loss converges to.
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score
X, y = load_digits(return_X_y=True)
# COMMON BUG: skipping the scaler. Pixel values are 0-16, so it's *mostly* fine,
# but on most real datasets unscaled features destroy PCA.
pipe = make_pipeline(
StandardScaler(),
PCA(n_components=0.95), # keep enough PCs for 95% variance
LogisticRegression(max_iter=1000),
)
scores = cross_val_score(pipe, X, y, cv=5)
print(f"Accuracy: {scores.mean():.3f} +/- {scores.std():.3f}")
print(f"Components kept: {pipe.named_steps['pca'].n_components_} (of 64)")
# Expected: ~0.94-0.96 accuracy with ~30 components (vs 64 raw features) — same accuracy, half the dimensions.
The two takeaways: (1) PCA inside an sklearn pipeline is one line and gets you free compression with no accuracy hit; (2) n_components=0.95 is the cleanest API — you ask sklearn for "enough PCs for 95% variance" and it figures out the number for you.
Dimensionality reduction trades fidelity for tractability. You give up some information in exchange for faster models, denoised features, and the ability to plot your data; the question is always how to pick which information to keep
PCA finds the directions of maximum variance via eigendecomposition of the covariance matrix — Center the data, build Σ, eigendecompose it, project onto the top-k eigenvectors; under the hood, sklearn uses SVD on the centered data because it's more numerically stable
The explained variance ratio is your knob. Pick k by demanding 90%, 95%, or 99% cumulative EVR; the elbow in the cumulative-EVR curve tells you the intrinsic dimensionality of your data
LDA is the supervised counterpart of PCA. When you have labels and Gaussian-ish classes, LDA finds directions that separate classes (between-class / within-class scatter ratio); capped at C-1 components for C classes
t-SNE and UMAP are for visualization only. They distort between-cluster distances and cluster sizes, are stochastic across runs, and (in t-SNE's case) don't transform new points; use them to see structure, not as feature extractors for downstream models
Autoencoders are PCA in deep-learning clothing. A linear autoencoder with MSE reconstruction converges to PCA; add non-linearities and you get a learned, invertible, transformable non-linear DR that scales to billions of parameters
Why does PCA require centering the data before computing the covariance matrix?
You can now compress 784-D images into 50 PCs that retain 90% of the signal, project them into 2D scatter plots that reveal class structure, and pick the right algorithm for each goal. Next, we shift from "how do we represent the data?" to "how do we trust the model?" — model interpretation, the techniques (SHAP, LIME, partial-dependence plots) that turn black-box ML predictions into stories a stakeholder can read.
Incremental PCA or Truncated SVD
Streaming-friendly variants of PCA
Data lies on a curved manifold (Swiss roll, ring, sphere)
Kernel PCA, Isomap, UMAP
Linear methods can't unfold curved structure
You want a learned non-linear embedding you can transform new points with
Autoencoder (next track)
Deterministic, invertible, generalizes; PCA in deep-learning clothing
You need interpretable factors that are sparse or non-negative
NMF, Sparse PCA
When loadings need to tell a story, not just compress