Eigenvalues & SVD

#The Hinge on the Door

This is one of the most beautiful ideas in all of math. It sounds intimidating, but the core concept is simple: every transformation has a few "special directions" that reveal its true nature. Once you see it, you will never look at data the same way again.

One honest caveat: some matrices have no such direction at all. A pure rotation, such as turning every vector by 90 degrees, has no real eigenvector ("real" here just means ordinary numbers), because every direction gets turned. The Going Further section at the end says what happens then.

#A First Test: Which Directions Do Not Turn

Before any formula, try one matrix on two vectors. Take $A = \begin 3 & 1 \ 0 & 2 \end$. To multiply it by a vector $[x, y]$, take row one times the vector for the top answer and row two times the vector for the bottom answer.

First test vector $[1, 0]$:

$A\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 3\cdot 1 + 1\cdot 0 \\ 0\cdot 1 + 2\cdot 0 \end{bmatrix} = \begin{bmatrix} 3 \\ 0 \end{bmatrix} = 3\begin{bmatrix} 1 \\ 0 \end{bmatrix}$

The answer points the same way as the input, just 3 times as long. Same direction, stretched by 3: this is an eigenvector, and 3 is its eigenvalue.

Second test vector $[0, 1]$:

$A\begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 3\cdot 0 + 1\cdot 1 \\ 0\cdot 0 + 2\cdot 1 \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$

The input pointed straight up; the answer leans to the right. The matrix turned it, so $[0, 1]$ is not an eigenvector.

Now try it yourself. Spin the test vector around and watch where the output lands. Look for the directions where the output stays on the same line as the input.

Those directions that do not turn are the eigenvectors, and the stretch factor along each one is its eigenvalue. The next section writes that idea as a single equation.

#The Core Equation

Read it as: "A times v lands on the same line as v, only $\lambda$ times as long." In the test above, $\vec = [1, 0]$ and $\lambda = 3$.

Try it! Open the Python REPL (bottom-right of the screen: click Quick Actions, then Python) and type these lines yourself.

Where:

• A is the matrix (the transformation)
• v is an eigenvector (a direction that survives the transformation unchanged)
• lambda is the eigenvalue (the stretch factor along that direction)

#Interactive: See Eigenvectors Hold Steady

Watch how an eigenvector keeps its direction while another vector turns. Apply the matrix and compare the two arrows:

The eigenvector's arrow gets longer or shorter but stays on its own line. The other arrow lands on a different line, which means it was turned.

#What Eigenvalues Tell You

Every row is a stretch, a shrink, or a flip along a line, never a turn. A matrix that turns every direction, like a pure rotation, has no real eigenvalue at all.

#Finding Eigenvalues

Guessing directions worked for $A = \begin 3 & 1 \ 0 & 2 \end$ because the numbers were friendly. Here is a method that always works for a 2x2 matrix. We take the same A and go one line at a time.

#Worked Example: Finding Eigenvalues by Hand

$A - \lambda I = \begin{bmatrix} 3-\lambda & 1 \\ 0 & 2-\lambda \end{bmatrix}$

$\det(A - \lambda I) = (3-\lambda)(2-\lambda) - 1\cdot 0 = (3-\lambda)(2-\lambda)$

Verify: $A\vec_1 = \begin 3 \ 0 \end = 3\begin 1 \ 0 \end$ ✓ and $A\vec_2 = \begin 3\cdot 1 + 1\cdot(-1) \ 0\cdot 1 + 2\cdot(-1) \end = \begin 2 \ -2 \end = 2\begin 1 \ -1 \end$ ✓

Any multiple of an eigenvector is also an eigenvector, so $[2, 0]$ works as well as $[1, 0]$. Only the direction matters. The first test in this lesson already found $[1, 0]$ by trial; the method just found it, and the second direction, by algebra.

#SVD: The Universal Decomposition

Its claim is that every matrix does three simple things in a row: turn, stretch, turn. Switch the three steps on one at a time and watch the circle turn, become an ellipse, then turn again:

Written as a formula, the three steps are:

Where:

• V^T ("V transpose", which means V with its rows and columns swapped, so the first column becomes the first row) turns the input into the "natural" directions of the transformation
• Sigma (capital Greek letter sigma) stretches along each axis by the singular values (never negative, sorted largest to smallest)
• U turns the result into the output directions

U and V are pure turns (possibly with a flip). A matrix like that is called orthogonal: it turns things without changing any length. Only Sigma changes lengths.

The first singular value captures the most important stretching; the last captures the least. If the last few singular values are near zero, those dimensions barely matter, and you can safely throw them away.

#Truncated SVD: Practical Compression

Here k is how many singular values you keep. Use the low-rank toggle in the next lab to keep fewer singular values and watch the point cloud change.

The data effectively lives in a 10-dimensional subspace, even though it is nominally 500-dimensional. The 490 near-zero singular values correspond to directions with almost no variation, so you can approximate the matrix well using only the top 10.

Here is how the saving is counted. Keeping k = 10 means storing three pieces. $U_k$ has 1000 rows and 10 columns: 10,000 numbers. $\Sigma_k$ is just the 10 singular values: 10 numbers. $V_k^T$ has 10 rows and 500 columns: 5,000 numbers. The total is 10,000 + 10 + 5,000 = 15,010 numbers, instead of 1000 x 500 = 500,000. That is about 33 times smaller (500,000 / 15,010 = 33.3) with minimal information loss. In general the count is k x (rows + columns + 1), which is 10 x (1000 + 500 + 1) = 15,010.

#Seeing It Work: Rebuilding an Image from k = 1, 2, 5

The cell below builds a small synthetic 30 x 30 image (a bright square plus a left-to-right brightness ramp, with a little seeded noise), then rebuilds it from only k singular values and prints how far each rebuild is from the original.

The real output: the first six singular values are 20.05, 4.32, 0.52, 0.46, 0.43 and 0.38. Two values stand far above the rest, because the image is mostly a square plus a ramp, and everything else is noise. With k = 1 you store 61 numbers instead of 900 and the relative error is 0.220. With k = 2 you store 122 numbers and the error drops to 0.065. With k = 5 you store 305 numbers and the error only falls to 0.052, because what is left is the noise. Past k = 2, more singular values buy almost nothing.

#Why SVD Matters for ML

#Try It Yourself

You have checked eigenvectors by guessing and by algebra. Now do the algebra yourself on a new matrix, [[4, 2], [1, 3]], and let Python check each step. For a 2x2 matrix, the quadratic from the characteristic equation always has the form $\lambda^2 - (\text)\lambda + \det$, where the trace is the sum of the diagonal.

The real output: the polynomial is $\lambda^2 - 7\lambda + 10$, so the eigenvalues are 5 and 2. For $\lambda = 5$ the eigenvector is $[2, 1]$ and $A\vec = [10, 5] = 5\cdot[2, 1]$. For $\lambda = 2$ the eigenvector is $[2, -2]$ and $A\vec = [4, -4] = 2\cdot[2, -2]$. The eigenvalues add to 7, which is the trace, and multiply to 10, which is the determinant. These two relationships hold for every square matrix. For the diagonal matrix B the eigenvalues are just the diagonal entries, 3 and 2.

⚡ Playground: Eigen Decomposition → — decompose a matrix and watch the singular vectors stretch the space.

#Run It Yourself: Eigen & SVD on a Real 3×3 Matrix

#Key Takeaways

• Eigenvectors are directions that survive unchanged. When a matrix is applied, eigenvectors only get stretched (not rotated), and the eigenvalue tells you the stretch factor along that direction
• Eigenvalues reveal a transformation's DNA. Large eigenvalues indicate amplified directions, small ones indicate compressed directions, and zero eigenvalues mean that dimension is destroyed entirely
• SVD decomposes any matrix into rotate-stretch-rotate. Unlike eigendecomposition, SVD works on any matrix (including non-square ones), making it the universal tool for understanding matrix transformations
• PCA is eigendecomposition of the covariance matrix (a table of how pairs of features vary together, covered properly later in the track). The eigenvectors with the largest eigenvalues capture the directions of maximum variance, enabling dimensionality reduction that preserves the most information
• LoRA exploits low-rank structure. Weight updates during fine-tuning live in a low-dimensional subspace, so a low-rank approximation (inspired by SVD) captures the essential update with far fewer parameters

#Quick Check

#Going Further (Optional)

Everything in this section is optional. The main path ends above: eigenvectors, eigenvalues, finding them for a 2x2 matrix, and SVD as rotate, stretch, rotate. Come back here when those feel comfortable.