"king − man + woman ≈ queen" sounds like a parlor trick. It's not — it's the single mathematical discovery that powers Google Search, Spotify's "Song Radio," GitHub Copilot's code search, and every RAG pipeline in production. Pick the wrong embedding model and your retrieval is dead before chunking even matters. This lesson is the foundation everything else in this track stands on.
Learning Objectives
After this lesson, you will be able to:
Understand how embedding models turn text into lists of numbers (vectors) where similar meanings end up close together in high-dimensional space
Calculate cosine similarity between two vectors and interpret the score as 'how related are these two pieces of text?' — and know why cosine beats Euclidean distance for text
Compare popular embedding models (OpenAI text-embedding-3, BGE, E5) and understand the trade-offs (speed vs. accuracy vs. cost vs. dimensionality) when picking one for your project
Explain how contrastive learning trains embedding models using positive and negative pairs, and why asymmetric query/document embeddings improve RAG retrieval
Embeddings are one of those ideas that sounds intimidating but is actually beautifully simple once it clicks. You are going to have a genuine "aha" moment in this lesson -- most people do.
Active Recall
Before we step into RAG: recall from the vectors-and-spaces lesson, write the formula for cosine similarity between two vectors a and b in one line. Then state in one sentence why we normalize by the vector magnitudes. Don't scroll back — commit your answer first. This formula is the entire mathematical engine of embedding retrieval; if it's not on the tip of your tongue, this lesson is the place to lock it in.
Write your answer in your own words — don't look back at the lesson. This is the most effective way to remember what you just learned.
Embeddings are this map. They convert text into coordinates in a high-dimensional space where proximity equals meaning. "The cat sat on the mat" and "A feline rested on the rug" end up at nearly the same coordinates, even though they share almost no words. "The stock market crashed" ends up far away from both.
This is the foundation of everything in RAG: the ability to measure how similar in meaning two pieces of text are, not just how similar their words are.
Traditional keyword search fails at understanding meaning. If your document says "automobile" and the user searches "car," keyword search misses it entirely. If a document discusses "treatment for hypertension" and the user asks about "lowering high blood pressure," keyword search sees zero overlap. Embeddings solve this by operating in meaning space rather than word space.
An embedding model takes a string of text and outputs a vector -- a list of floating-point numbers, typically between 256 and 3072 dimensions. Each dimension captures some aspect of meaning, though individual dimensions are not human-interpretable.
embed("The cat sat on the mat")=[0.023,−0.187,0.442,…,0.091]∈Rd
Modern embedding models are trained using contrastive learning. The training process goes like this:
Positive pairs: Take pairs of text that should be similar (a question and its correct answer, a sentence and its paraphrase, a query and a relevant document).
Negative pairs: Take pairs that should not be similar (a question paired with an irrelevant document).
Train the model to produce vectors that are close for positive pairs and far apart for negative pairs.
L=−logesim(q,d+)/τ+∑d−esim(q,d−)/τesim(q,d+)/τ
After training on millions of such pairs, the model learns to map semantically similar text to nearby points in vector space, regardless of the specific words used.
Try it! Go to projector.tensorflow.org and type words like "king," "queen," "man," and "woman" into the search. Watch how related words cluster together in 3D space. Rotate the visualization and notice the geometric patterns -- this is exactly what embedding models learn to produce.
Try it: Explore word similarities in embedding spaceInteractive
If 'king' - 'man' + 'woman' = ?, what vector would you expect to be closest to the result?
One of the most remarkable properties of embeddings is that semantic relationships become geometric relationships. The classic example: the vector for "king" minus "man" plus "woman" produces a vector closest to "queen." The model has learned that the relationship between king and queen is the same as the relationship between man and woman -- a direction in vector space that represents gender.
This works for many relationships:
Paris - France + Japan = Tokyo (capital-of relationship)
walked - walk + swim = swam (past-tense relationship)
bigger - big + small = smaller (comparative relationship)
These are not programmed rules. They emerge naturally from the geometry of the learned embedding space.
A piece of text enters the embedding model. It can be a single word like "king," a sentence, or an entire paragraph. The model needs to convert this raw text into a numerical representation that captures its meaning.
The tokenizer splits the text into tokens. Each token is mapped to a row in a learned embedding matrix -- a giant table of numbers with one row per vocabulary item and one column per dimension. For "king," the model retrieves the row corresponding to that token.
The model produces a dense vector -- a list of floating-point numbers like [0.2, -0.5, 0.8, 0.1, ...]. For modern models, this is 768 to 3072 numbers. Each dimension encodes some abstract aspect of meaning, though no single dimension is human-interpretable on its own.
Words with similar meanings end up at nearby coordinates in this high-dimensional space. "King" is close to "monarch," "ruler," and "sovereign." It is far from "bicycle" or "tomato." The distance between points reflects semantic relatedness.
Remarkably, you can do algebra on these vectors. king - man + woman produces a vector that lands closest to queen. The model has learned that the direction from "man" to "woman" is the same as the direction from "king" to "queen" -- a gender relationship encoded as a geometric direction.
To quantify how similar two embeddings are, compute the cosine of the angle between them. A cosine similarity of 1.0 means identical meaning. Near 0 means unrelated. This single number powers every similarity search in RAG -- it is how the system decides which documents are relevant to a query.
Mikolov et al. at Google trained shallow neural networks to predict a word from its context (or vice versa). The hidden layer weights became the embeddings. For the first time, word relationships became geometric: king - man + woman = queen.
Limitation: One vector per word. "Bank" (financial) and "bank" (river) have the same embedding. No sentence-level understanding.
Reimers and Gurevych fine-tuned BERT with a Siamese network structure to produce meaningful sentence embeddings. For the first time, you could embed entire sentences and compare them meaningfully with cosine similarity.
Limitation: Fixed training data. Performance degraded on out-of-domain text.
Models like E5 and GTE introduced instruction-tuned embeddings. You prefix the text with a task instruction: "Represent this query for retrieving relevant documents:" vs "Represent this document for retrieval:". The model produces different embeddings for the same text depending on the task, dramatically improving RAG retrieval quality.
Embeddings are not limited to text. Modern models embed images, audio, and video into the same vector space as text:
CLIP (OpenAI): Embeds images and text in a shared space. "A photo of a cat" and an actual photo of a cat have similar vectors.
ImageBind (Meta): Extends to 6 modalities: text, image, audio, depth, thermal, and IMU data.
Multimodal RAG: Index images alongside text. A query "diagram showing neural network architecture" retrieves relevant diagrams, not just text descriptions.
Tests · Verify cosine similarity between 'cat on mat' and 'feline on rug' is > 0.95. Verify similarity between 'cat on mat' and 'stock market' is < 0.5.
Word-vector analogies are real, but they have subtleties. The playground below lets you build a tiny embedding space from scratch, run the king − man + woman experiment numerically, and explore Matryoshka truncation (the 2024 trick that lets you store one 1536-d vector and search at 64, 256, 512, or 1536 dims depending on latency budget).
Embeddings convert text into dense vectors that capture semantic meaning. Similar concepts map to nearby points in vector space, enabling "semantic search" that finds relevant content even when exact keywords do not match
Cosine similarity measures semantic relatedness. It compares the angle between vectors regardless of magnitude, giving a score from -1 (opposite meaning) to +1 (same meaning) that powers retrieval ranking
Embedding model choice significantly impacts RAG quality. Larger models produce better embeddings but are slower and more expensive; the right model depends on your domain, latency requirements, and accuracy needs
Embeddings are the foundation of modern information retrieval. Every vector database, semantic search engine, and RAG system depends on high-quality embeddings to bridge the gap between natural language queries and stored knowledge
A dense vector (typically 256-3072 dimensions) produced by a neural network that encodes the semantic meaning of a piece of text, image, or other input.
A vector where most components are non-zero and each dimension carries continuous semantic information. Contrasts with sparse representations like TF-IDF or bag-of-words.
Training regime used for modern embedding models. Positive pairs (query and relevant doc) are pulled together in vector space while negative pairs are pushed apart, typically via the InfoNCE loss.
Angle-based similarity metric: a . b / (||a|| * ||b||). Returns a value in [-1, 1] and is the default ranking function for text embeddings because it ignores magnitude.
Architecture that encodes query and document independently into vectors, then compares via dot product or cosine. Fast and scalable — the standard for first-stage retrieval.
Architecture that takes query and document as a joint input and outputs a single relevance score. More accurate than a bi-encoder but ~100x slower; used for re-ranking the top candidates.
Embedding where the first N dimensions are themselves a valid lower-dimensional embedding. Lets you run fast search on truncated vectors and re-rank with the full vector — huge storage/speed wins at scale.
Embedding models (Cohere embed-v3, E5) that use different prompts/modes for queries vs. documents, producing vector spaces optimized for retrieval rather than symmetric similarity.
Where This Matters
Notion
Semantic Search at Notion AI
Notion AI embeds every block across a workspace so users can ask questions in natural language and retrieve relevant notes by meaning, even when no keywords overlap.
↑
Turns millions of docs per workspace into a searchable knowledge base
OpenAI
CLIP Multimodal Embeddings
CLIP embeds images and text into a shared vector space, enabling image-to-text and text-to-image search. 'A photo of a cat' and an actual cat photo land at nearly the same coordinates.
↑
Foundational model behind Stable Diffusion, DALL-E, and most image search systems
Glean / Perplexity Enterprise
RAG over Private Docs at Enterprises
Enterprises index Slack, Confluence, Drive, Jira, and email as embeddings so employees can ask 'what did we decide about X?' and retrieve the actual source passages as context for an LLM.
↑
Cuts time-to-answer for internal questions from hours to seconds
Interview Practice
You now understand how text becomes vectors and how similarity is measured. But with millions of vectors, how do you search efficiently? Next up: Vector Databases & Indexing -- the data structures that make similarity search possible at scale.