Every word in ChatGPT is a 12,288-dimensional vector. Every image in Midjourney is a 512-dimensional vector. Every recommendation Netflix makes is a dot product. Linear algebra is the language ML is written in — and it's simpler than it looks. If you can add two numbers, you already know 80% of what's coming.
Learning Objectives
After this lesson, you will be able to:
Understand vectors as arrows with a direction and a length -- not just lists of numbers
Calculate dot products and cosine similarity, and explain why they measure how similar two things are
See that every dataset in machine learning is really just a collection of these number-arrows living in a space with many dimensions
Understand basis vectors and span, and explain why the effective dimensionality of data is often much smaller than its nominal number of dimensions
What Is a Vector, Really?
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
Math is not a talent -- it is a skill. If you can count, add, and follow directions on a map, you already have everything you need to understand vectors. This lesson starts from zero and builds up one piece at a time. Take it slow and you will surprise yourself.
This lesson assumes zero math background. If you can add numbers, you can learn vectors. By the end, you will understand how every piece of data in machine learning is represented -- and why.
In mathematics, we write a 2D vector as a column of numbers:
v=[34]
Try it! Open the Python REPL (bottom-right of the screen: click Quick Actions, then Python) and type these lines yourself.
The top number (3) says "move 3 units along the x-axis." The bottom number (4) says "move 4 units along the y-axis." Together, they define a direction and a distance.
Nothing changes conceptually when we jump from 2D to 100D or even 768D. A vector in 768-dimensional space is still just a set of instructions: "move this much along axis 1, this much along axis 2, ... this much along axis 768."
You cannot visualize 768 dimensions, and that is perfectly fine. The math works exactly the same way. The magnitude formula just gets more terms under the square root.
Multiplying a vector by a number (a scalar) stretches or shrinks it without changing direction. Multiply by -1 and it flips to point the opposite way.
2⋅[34]=[68]
What Do You Think?
If you multiply the vector [3, 4] by -0.5, what happens geometrically?
Multiplying by -0.5 does two things: the 0.5 halves the length, and the negative sign flips the direction. The result is [-1.5, -2] -- a vector half as long, pointing in the opposite direction.
To make the connection between vectors and operations concrete, here is an interactive sandbox. Drag the head of vector a or b and watch the addition and scalar multiplication update in real time.
Loading visualization...
Quick check
A vector v has magnitude 10. You multiply it by the scalar -2. What are the magnitude and direction of the result?
The dot product is the single most important operation in all of ML. It answers a simple question: how much do these two vectors point in the same direction?
The dot product has a beautiful geometric meaning:
a⋅b=∥a∥∥b∥cosθ
When you normalize both vectors to length 1, the dot product IS the cosine of the angle between them. This gives us cosine similarity, used in everything from search engines to word embeddings:
cosine_similarity(a,b)=∥a∥∥b∥a⋅b
In code, this is torch.matmul(Q, K.transpose(-2, -1)) / sqrt(d_k) — the first line of every transformer's attention computation. The Q@K^T matrix is just a batch of dot products.
Now compute it for real. The playground below runs actual numpy in your browser — no setup, no install. Edit the vectors, click Run, and watch the dot product, magnitudes, and cosine similarity update.
Everything starts as raw data -- image pixels, text tokens, audio samples, or spreadsheet rows. At this stage, the data is not yet in a form that any ML model can work with. A 28x28 grayscale image is just a grid of brightness values. A sentence is just a sequence of characters.
The raw data is converted into vectors -- lists of numbers that live in a high-dimensional space. A 28x28 image becomes a 784-dimensional vector. A word like "king" becomes a 768-dimensional embedding vector. This is the critical step: vectorization maps messy real-world data into the precise mathematical world where ML algorithms operate.
Once data is vectorized, ML algorithms use vector operations to extract meaning. The dot product measures how similar two vectors are. Cosine similarity normalizes for magnitude. Matrix multiplications (which are batches of dot products) transform vectors through neural network layers. The visualization on the right shows how a matrix warps the entire vector space.
The model processes these vectors through layers of transformations. In a transformer, query and key vectors are compared via dot products to compute attention. In a recommendation system, user and item vectors are compared via cosine similarity. In RAG, document and query vectors are compared to find relevant context. Every ML operation ultimately reduces to vector math.
Play with this visualization to build intuition for how vectors transform:
Try it: Drag vectors and watch transformationsInteractive
Loading visualization...
Try this: Set the matrix to the identity [[1,0],[0,1]] and watch how vectors stay unchanged. Then try [[2,0],[0,2]] -- everything doubles. Then try [[0,-1],[1,0]] -- everything rotates 90 degrees. Each matrix defines a different transformation of the entire vector space.
A vector space is the arena where vectors live. It is defined by a set of objects (vectors), two operations (addition and scalar multiplication), and rules these operations must follow.
The most common vector space in ML is R^n -- the set of all n-tuples of real numbers. When we say "a 768-dimensional vector," we mean a point in R^768.
Any vector in a space can be built by combining a set of basis vectors. In 2D, the standard basis is:
e^1=[10],e^2=[01]
The vector [3, 4] is just 3 times e1 plus 4 times e2. This is called a linear combination. The span of a set of vectors is every vector you can create through linear combinations. If two vectors are not parallel, they span the entire 2D plane. If they are parallel, they only span a line.
This idea is crucial: the rank of your data matrix tells you how many truly independent dimensions your data occupies. Data that nominally lives in 784D often occupies a much smaller subspace -- which is exactly what dimensionality reduction exploits.
Quick check
You have three vectors in 3D: v1 = [1, 0, 0], v2 = [0, 1, 0], v3 = [2, 3, 0]. What is the dimension of their span?
Vectors are geometric objects. They have direction and magnitude, not just lists of numbers; every data point in ML is represented as a vector in high-dimensional space
Dot product measures alignment. It quantifies how much two vectors point in the same direction, making it the core operation behind similarity search, attention mechanisms, and recommendation systems
Cosine similarity normalizes for magnitude. By dividing the dot product by both vector lengths, cosine similarity measures pure directional alignment regardless of scale, which is why it powers embedding-based search
Basis vectors and span define the space. Any vector can be built from linear combinations of basis vectors, and the effective dimensionality of your data (its rank) is often much smaller than the nominal number of dimensions
High dimensions work the same way. The math for 2D vectors extends identically to 768D or higher; you cannot visualize it, but the operations (dot product, magnitude, cosine similarity) are unchanged
What does a dot product of zero between two vectors tell you?
Key Terms8 terms
An ordered list of numbers that represents a point (or displacement) in n-dimensional space; it has both a direction and a magnitude.
A single real number with no direction. Multiplying a vector by a scalar stretches or shrinks it without changing its direction (unless the scalar is negative).
The length of a vector, computed as the square root of the sum of its squared components. Denoted ||v|| or sometimes |v|.
A scalar result computed by multiplying corresponding components of two vectors and summing them. Measures how aligned two vectors are.
The dot product normalized by both vector magnitudes, giving a value in [-1, 1] that measures only angular alignment, ignoring length.
Two vectors are orthogonal (perpendicular) when their dot product is zero. In ML, orthogonal (mean-centred) features are uncorrelated: neither linearly predicts the other. Uncorrelated is weaker than independent.
A set of linearly independent vectors that span a space. Every vector in the space can be uniquely expressed as a linear combination of the basis vectors.
The set of all vectors you can produce by linear combinations of a given set of vectors. In 2D, two non-parallel vectors span the entire plane.
Where This Matters
OpenAI
Semantic Search at OpenAI
OpenAI's text-embedding-3 models turn documents into 1536-dim vectors so retrieval systems can rank results by cosine similarity instead of keyword overlap.
↑
Enables meaning-based search across billions of documents
Spotify
Music Recommendations at Spotify
Every track becomes a vector of learned audio and taste features. 'Discover Weekly' finds songs whose vectors sit closest to your listening history in that space.
↑
Drives 30%+ of total streaming through personalized discovery
Meta
Visual Search at Meta / Pinterest
Images are encoded as vectors by a vision backbone; 'visually similar' lookups use nearest-neighbor search in that vector space to surface matching content and ads.
↑
Powers shop-the-look and reverse image search at scale
Interview Practice
Next up: we will see how matrices transform these vectors -- warping, rotating, and scaling entire vector spaces. Every neural network layer does exactly this.