"In a 1,000-dimensional space, every two random points are roughly the same distance apart. That single fact — Bellman's 'curse of dimensionality' — explains why your k-NN model gets weirder as you add features, why your clustering stops separating, and why every modern ML system ultimately learns or imposes a low-dimensional structure on its inputs. Compression isn't a luxury; it's how learning works."
Learning Objectives
After this lesson, you will be able to:
Explain the curse of dimensionality — why distances flatten out and models struggle when you have hundreds or thousands of features
Apply PCA to compress high-dimensional data into a small set of orthogonal directions that capture most of the variance, and read a scree plot to pick the right number of components
Choose between PCA, t-SNE, and UMAP based on the goal: PCA for compression that feeds downstream models, t-SNE/UMAP for visualization, and autoencoders for nonlinear reconstruction
Avoid the four classic mistakes — running PCA without standardizing, using t-SNE coordinates as features, fitting on the full dataset before splitting, and over-interpreting axes that don't have physical meaning
Build this --> Take the MNIST digits dataset (each image is a 784-dim vector), reduce it to 2-D with PCA and t-SNE and UMAP, color the points by digit label, and watch which method makes the digits separate most cleanly — you will instantly see why nonlinear methods are popular for visualization
Don't worry if "compress 1000 features into 10" sounds like magic — by the end of this lesson, you will have a clear mental model of what gets compressed, what gets thrown away, and when that trade is worth making.
The fundamental question of dimensionality reduction: can we describe a high-dimensional dataset using a much smaller number of variables, while preserving the structure that matters? For most real datasets the answer is yes — features are correlated, the data lives on a low-dimensional manifold embedded in the high-dimensional space, and we just need the right tool to find it.
#Linear Method 1: Principal Component Analysis (PCA)
PCA finds new axes for your data such that:
Axis 1 points in the direction of maximum variance in the data.
Axis 2 points in the direction of maximum variance orthogonal to axis 1.
Axis 3 is orthogonal to both, etc.
Each new axis is a principal component. By keeping only the top-k components, you compress an n-dimensional dataset into k dimensions while losing the least possible variance.
The explained variance ratio of component i is its eigenvalue divided by the sum of all eigenvalues — the fraction of total variance captured by that axis.
EVRi=∑j=1dλjλiCumulativek=i=1∑kEVRi
Common heuristics for choosing k
95% variance rule: keep enough components to retain 95% of the total variance.
Elbow rule: visually find where the scree curve bends from steep to flat.
Kaiser criterion: keep components with eigenvalue > 1 (only meaningful when features are standardized).
Cross-validate: treat k as a hyperparameter — pick the k that maximizes downstream model performance.
Try it: Project a high-dimensional dataset and see how each component captures varianceInteractive
PCA assumes the data's important structure is captured by linear directions of high variance. It struggles when:
The manifold is curled. A swiss-roll dataset is 2-D in nature but PCA can only flatten it, not unroll it.
High variance ≠ informative. PCA might pick noise that happens to be loud over signal that is subtle.
Classes overlap in variance. A direction that maximizes variance might not separate your classes — LDA does that, not PCA.
That is why we need nonlinear methods.
#Linear Cousins: Truncated SVD and Random Projection
Truncated SVD is PCA's twin for sparse matrices. It does not require centering the data (which would destroy sparsity), so it is the default for compressing TF-IDF vectors of text or one-hot encoded matrices.
Random Projection is the surprise tool that should be in your kit. The Johnson–Lindenstrauss lemma proves that you can project n points from d dimensions down to roughly O(log n / ε²) dimensions using a random Gaussian matrix and pairwise distances will be approximately preserved (within ε). It is dramatically faster than PCA and works astonishingly well as a preprocessing step before k-NN on giant datasets.
t-SNE answers a different question than PCA. PCA asks "what direction has the most variance?" t-SNE asks "which points are neighbors of which other points, and how can I draw them in 2-D so the neighborhoods are preserved?"
In high-D, define probability P_ij that point j is a neighbor of point i, using a Gaussian whose width depends on a hyperparameter called perplexity (intuitively, "expected number of neighbors", typically 5–50).
In low-D (2-D), find positions for the points such that the matching probability Q_ij — using a heavier-tailed Student-t distribution — is as close as possible to P_ij in KL-divergence sense.
Lt-SNE=i=j∑PijlogQijPij
Try it: Tune perplexity and watch t-SNE reveal cluster structureInteractive
The Distill.pub article "How to Use t-SNE Effectively" (Wattenberg, Viégas & Johnson, 2016) is required reading; the highlights:
Cluster sizes mean nothing. t-SNE expands tight clusters and shrinks loose ones to fit them on the page.
Distances between clusters mean nothing. Two t-SNE blobs sitting far apart are not further apart in the original space than two blobs sitting close.
Different runs give different layouts. Always set random_state and run multiple seeds before drawing conclusions.
Perplexity changes the picture. A perplexity of 5 emphasizes local micro-clusters; 50 reveals broad structure; the "right" answer is to look at several.
Random noise can look clustered. Even uniformly random high-D points produce blob-like t-SNE plots.
UMAP solves t-SNE's main problems while keeping the visual quality:
Faster. 5-10x speedup on large datasets due to efficient nearest-neighbor approximation.
Preserves global structure better. Inter-cluster distances on a UMAP plot are more interpretable than on a t-SNE plot.
Has a transform method. You can fit UMAP on training data and project new points without retraining.
Scales to millions of points. Single-cell genomics routinely runs UMAP on 1M+ cells.
UMAP is built on a graph approximation of the data manifold using fuzzy simplicial sets (formal mathematical machinery) — but for practical purposes you tune two knobs: n_neighbors (smaller = local structure, larger = global structure; default 15 is usually fine) and min_dist (how tightly to pack neighbors in the embedding; smaller = tighter clusters).
What Do You Think?
You have 1000-dim word embeddings for 10,000 words and you want to make a 2-D plot for a blog post. Which method should you reach for first?
PCA tends to produce a featureless blob for word embeddings because the meaningful structure is highly nonlinear. t-SNE works but is slow and distorts global distances. UMAP has been the practical default since ~2020 — it is fast, preserves both local clusters and broad structure, and you can transform new words into the same 2-D space later. (Random projection preserves distances but does not separate clusters into pretty pictures — it is for downstream models, not for plots.)
Kernel PCA applies the kernel trick to PCA: implicitly map data into a high-dimensional feature space (via RBF, polynomial, or sigmoid kernels), then run PCA there. It can unroll a swiss roll. It does not scale beyond a few thousand points because the kernel matrix is n×n.
Autoencoders are neural networks that learn a compressed code by training to reproduce their input through a narrow bottleneck. A 784→64→16→64→784 autoencoder learns a 16-D representation of MNIST digits. They are the modern, scalable, fully nonlinear answer to "compress AND reconstruct" — you will meet them properly in the Deep Learning track.
Tests · Verify Z_pca and Z_tsne are both shape (1797, 2). Confirm cumulative variance is monotonically increasing. The 1-NN accuracy on the 64-D original should be >= the 2-D projections.
High-dimensional space punishes you in three ways. Points become sparse, distances flatten out, and overfitting becomes inevitable; reducing dimensions is one of the cheapest cures
PCA finds the directions of maximum variance by computing eigenvectors of the covariance matrix; it is fast, deterministic, has a transform method for new data, and assumes linear structure
Always standardize before PCA unless every feature is in the same physical units; otherwise the largest-variance feature dominates the components
t-SNE and UMAP are visualization tools, not feature extractors. Their objectives distort global distances on purpose; UMAP has replaced t-SNE as the practical default since 2020
Match the method to the goal. PCA to compress for downstream models, UMAP/t-SNE to visualize, autoencoders to compress nonlinearly, random projection when speed matters more than quality
You forgot to standardize your features before running PCA. What is the most likely consequence?
The cleanest way to internalize PCA is to run it on a dataset you already understand and look at how much variance each component captures. The playground below fits PCA on the Iris dataset.
Loading visualization...
Compress smart, visualize honestly. Next up: Time Series Data — how to prepare data where every row is timestamped and the order matters as much as the values.