Google's PageRank is one eigenvector calculation. Netflix's recommendations use SVD. LoRA — the technique that lets you fine-tune a billion-parameter LLM on a laptop — is literally a low-rank SVD approximation. The big idea is simple: every matrix, no matter how complex, is secretly just rotate → stretch → rotate. Find those three pieces and you've cracked the matrix open.
Learning Objectives
After this lesson, you will be able to:
Identify eigenvectors as the special directions that a matrix only stretches (never rotates), and eigenvalues as how much it stretches them
Break any matrix apart into a simple sequence of rotate-stretch-rotate steps (called SVD) and explain why every matrix can be broken down this way
See how these ideas power real applications: compressing data, recommending songs, and customizing AI models on a laptop
Use truncated SVD to approximate a matrix with far fewer numbers, and calculate the compression ratio for a realistic example
Before we start: without looking back, write one sentence describing what a matrix DOES to a vector geometrically. (Hint: it's some combination of two simple operations.) The point isn't to be exactly right — it's to commit to an answer before you read the next paragraph.
Write your answer in your own words — don't look back at the lesson. This is the most effective way to remember what you just learned.
This is one of the most beautiful ideas in all of math. It sounds intimidating, but the core concept is simple: every transformation has a few "special directions" that reveal its true nature. Once you see it, you will never look at data the same way again.
Most matrices, when applied to a random vector, change both its direction and its length. But for many matrices, there exist special directions where the output vector points the same way as the input -- it just gets scaled. These special directions are eigenvectors, and the scaling factors are eigenvalues.
Try it! Open the Python REPL (bottom-right of the screen: click Quick Actions, then Python) and type these lines yourself.
Where:
A is the matrix (the transformation)
v is an eigenvector (a direction that survives the transformation unchanged)
lambda is the eigenvalue (the stretch factor along that direction)
What Do You Think?
If a 2x2 matrix has eigenvalues 3 and 0.5, what happens to a circle of points when you apply the matrix?
A circle becomes an ellipse. The eigenvectors define the axes of the ellipse, and the eigenvalues define how long each axis is. Eigenvalue 3 stretches one axis by 3x. Eigenvalue 0.5 shrinks the other to half. This is the geometric essence of eigendecomposition: every symmetric matrix transformation, when viewed through its own natural axes (eigenvectors), is just stretching.
Watch how eigenvectors maintain their direction while other vectors rotate. The colored arrows that do not change direction are the eigenvectors:
Eigenvector vs Random Vector — Apply the Matrix and See the DifferenceInteractive
Loading visualization...
Full Matrix Transform PlaygroundInteractive
Loading visualization...
Try this: Set the matrix to [[2,0],[0,3]]. The eigenvectors are along the x and y axes. Now try [[2,1],[0,3]] -- the eigenvectors shift. Notice how the colored eigenvector arrows get longer or shorter but never rotate.
The viz below goes one step further — it shows the full eigendecomposition A = QΛQ⁻¹. Edit the matrix and watch the eigenvalues, eigenvectors, and the diagonalization update in real time:
Not every matrix has nice eigenvectors. Non-square matrices do not even have eigenvalues in the traditional sense. SVD (Singular Value Decomposition) works on any matrix -- any shape, any content. It is the Swiss Army knife of linear algebra.
A=UΣVT
Where:
V^T rotates the input into the "natural" coordinate system of the transformation
Sigma stretches along each axis by the singular values (always non-negative, sorted largest to smallest)
U rotates the result into the output coordinate system
The first singular value captures the most important stretching; the last captures the least. If the last few singular values are near zero, those dimensions barely matter -- and you can safely throw them away.
You perform SVD on a 1000x500 matrix and find that the first 10 singular values are large while the remaining 490 are near zero. What does this tell you?
Quick check
For the symmetric matrix A = [[2, 0], [0, 5]], without computing anything, what are its eigenvalues and eigenvectors?
The data effectively lives in a 10-dimensional subspace, even though it is nominally 500-dimensional. The 490 near-zero singular values correspond to directions with almost no variation. You can approximate the matrix well using only the top 10 singular values -- reducing storage from 500,000 entries to roughly 10 x (1000 + 500 + 10) = 15,100 entries. That is a 33x compression with minimal information loss.
When you load a 7B-parameter model with peft and pass r=8, the library is performing a learned rank-8 SVD of the weight update. The "r" is exactly the number of singular values you keep.
#Condition Number: Why Some Matrices Are Numerically Dangerous
Singular values do more than tell you about compression -- they also tell you how numerically stable a matrix is. The condition number of a matrix is the ratio of its largest singular value to its smallest:
κ(A)=σminσmax
#Positive Semi-Definite Matrices: The Friendly Family
A square matrix A is positive semi-definite (PSD) if x^T A x ≥ 0 for every vector x (and positive definite if the inequality is strict for nonzero x). PSD matrices are the well-behaved citizens of linear algebra: their eigenvalues are real, non-negative, and their eigenvectors form an orthogonal basis.
A⪰0⟺xTAx≥0∀x
The most important PSD matrices in ML are covariance matrices (Σ = E[(x − μ)(x − μ)^T]) and Gram matrices (A^T A for any A). For any data matrix X, the matrix X^T X is automatically PSD because x^T (X^T X) x = ‖X x‖² ≥ 0. This is the algebraic reason PCA always returns real, non-negative eigenvalues that you can sort cleanly from largest to smallest.
For a PSD matrix, the eigenvalues and singular values coincide. For a positive definite matrix (no zero eigenvalues), the matrix is invertible and the determinant is the product of all eigenvalues. Most kernel methods (SVMs, Gaussian processes) and regularization tricks (Cholesky decomposition, the matrix square root used in whitening) require their input matrices to be PSD; if numerical noise makes a covariance matrix slightly indefinite (a tiny negative eigenvalue), production code typically projects back onto the PSD cone by clipping eigenvalues at zero or adding ε I.
Tests · Verify that A * v1 equals lambda1 * v1 and A * v2 equals lambda2 * v2. Check that det(A) = lambda1 * lambda2.
Matrix Computations
Gene Golub, Charles Van Loan (2013)
The definitive reference on numerical linear algebra including SVD algorithms. Chapter 2 covers eigenvalue problems; Chapter 8 covers SVD in depth. The algorithms in this book run inside every call to numpy.linalg.
⚡ Playground:Eigen Decomposition → — decompose a matrix and watch the singular vectors stretch the space.
#Run It Yourself: Eigen & SVD on a Real 3×3 Matrix
Time to verify everything by running real numpy.linalg.eig and numpy.linalg.svd in your browser. Try changing the matrix to something asymmetric and notice that eigenvalues can go complex — but singular values are always real and non-negative.
Loading visualization...
Active Recall
Close this lesson tab in your head for 30 seconds and write: (a) what the equation Av = λv MEANS in one English sentence, (b) why a 2D rotation matrix has no real eigenvectors, and (c) what 'r' represents when you load a LoRA adapter with r=8. If any of the three is fuzzy, that is the part that has not stuck yet — note it and come back to it tomorrow. Retrieval beats re-reading.
Write your answer in your own words — don't look back at the lesson. This is the most effective way to remember what you just learned.
Eigenvectors are directions that survive unchanged. When a matrix is applied, eigenvectors only get stretched (not rotated), and the eigenvalue tells you the stretch factor along that direction
Eigenvalues reveal a transformation's DNA. Large eigenvalues indicate amplified directions, small ones indicate compressed directions, and zero eigenvalues mean that dimension is destroyed entirely
SVD decomposes any matrix into rotate-stretch-rotate. Unlike eigendecomposition, SVD works on any matrix (including non-square ones), making it the universal tool for understanding matrix transformations
PCA is eigendecomposition of the covariance matrix. The eigenvectors with the largest eigenvalues capture the directions of maximum variance, enabling dimensionality reduction that preserves the most information
LoRA exploits low-rank structure. Weight updates during fine-tuning live in a low-dimensional subspace, so a low-rank approximation (inspired by SVD) captures the essential update with far fewer parameters
What does it mean if a matrix has an eigenvalue of zero?
Next up: Derivatives and Gradients -- the mathematical engine behind backpropagation. You will learn how a tiny nudge to an input ripples through a function to change the output, and why this is the single most important idea in training neural networks.