A machine learns that "king" − "man" + "woman" ≈ "queen" — purely from reading text. Word embeddings turn discrete tokens into dense vectors where geometry equals meaning, and the same principle scales all the way up to the 12,288-dimensional embeddings inside GPT-4 and the input layers of Claude Sonnet 4.6 and Llama 3.3. This is one of the most beautiful ideas in all of AI.
Learning Objectives
After this lesson, you will be able to:
Understand why neural networks need words turned into numbers before they can do anything useful
See why the simplest approach (one-hot encoding) breaks down for real language
Learn how Word2Vec discovers word meaning automatically using two clever tricks (CBOW and Skip-gram)
Understand how GloVe figures out word meaning by counting which words appear near each other
See how the embedding layer in neural networks feeds into transformers
Measure how similar two words are using cosine similarity
Neural networks only understand numbers. Feed them the word "cat" and they stare blankly. We need a way to convert words into numerical representations -- and not just any representation, but one that captures meaning. The word "cat" should be numerically similar to "kitten" and different from "tractor".
This lesson covers the journey from naive word representations to the powerful embeddings that underpin every modern language model.
This seems clean, but it has three fatal problems:
Massive dimensionality. A real vocabulary has 50,000+ words. Each vector is 50,000-dimensional with a single non-zero entry. That is extraordinarily wasteful.
No similarity information. The dot product between any two different one-hot vectors is always zero. "Cat" is exactly as similar to "dog" as it is to "democracy" -- which is absurd. The representation encodes no meaning whatsoever.
No generalization. If the model learns something about "cat" (e.g., it is an animal, it is a pet), that knowledge does not transfer to "kitten" at all because their representations share nothing.
cat⋅dog=0,cat⋅democracy=0
We need dense vectors -- short (say 300 dimensions), where every element carries information, and similar words have similar vectors.
Try it! Pick any two words -- say "happy" and "sad." Now imagine them as dots on a map. Would they be close together (similar meaning) or far apart? What about "happy" and "toaster"? You are already doing embedding intuition in your head. Keep this spatial picture as you read on.
In 2013, Tomas Mikolov at Google proposed a brilliant insight: a word's meaning is defined by the company it keeps. If "cat" and "dog" appear in similar contexts ("The ___ sat on the mat", "I fed the ___"), they should have similar vector representations.
Word2Vec trains a shallow neural network on a massive text corpus using one of two strategies:
Given a center word, predict the surrounding context words. This is the reverse of CBOW.
P(wt−2,wt−1,wt+1,wt+2∣wt)
Skip-gram creates more training examples per word (one for each context position), so it captures better representations for rare words.
What Do You Think?
Which two words would be closest in embedding space: cat/dog, cat/democracy, or cat/refrigerator?
The answer is cat/dog. Both words appear in contexts about animals, pets, and homes. Word2Vec learns this automatically -- no one tells the model that cats and dogs are both animals. The model discovers it from patterns in billions of sentences.
The most famous result from Word2Vec is that learned embeddings encode semantic relationships as vector directions:
king−man+woman≈queen
This works because the model learns that the direction from "man" to "woman" is the same as the direction from "king" to "queen" -- both encode the concept of gender. Similarly:
Paris - France + Italy ≈ Rome (capital city relationship)
In 2014, Stanford's Jeffrey Pennington, Richard Socher, and Christopher Manning proposed GloVe (Global Vectors for Word Representation) -- a different approach that learns from global word co-occurrence statistics.
The idea: build a giant matrix counting how often each word appears near every other word across the entire corpus. Then factorize this matrix into dense vectors.
wiTw~j+bi+b~j=log(Xij)
GloVe combines the best of both worlds: it uses global statistics (like traditional count-based methods) but learns dense vectors (like Word2Vec). In practice, GloVe and Word2Vec produce embeddings of similar quality.
Word2Vec vs GloVe -- which is better? Neither consistently wins. Word2Vec is a prediction-based method (predict context from word or vice versa). GloVe is a count-based method (factorize the co-occurrence matrix). Both produce embeddings that capture similar semantic relationships. The choice often comes down to practical considerations like training speed and corpus size.
Modern neural networks do not use pre-trained Word2Vec or GloVe directly. Instead, they include a learnable embedding layer as the very first layer of the network.
Create a matrix of shape [vocabulary_size x embedding_dim]. For a 50,000-word vocabulary with 512-dimensional embeddings, that is a 50,000 x 512 matrix. Each row corresponds to one word's embedding vector. Initially, all values are random.
When a word enters the model, its token ID is used as an index to look up the corresponding row in the embedding table. The word "cat" (token ID 3421) retrieves row 3421 from the matrix. This lookup is just a table indexing operation -- no multiplication needed.
As the full model trains on its task (translation, classification, generation), gradients flow back into the embedding table. Each word's vector is nudged to better serve the downstream task. Words that behave similarly in context gradually develop similar vectors.
Unlike generic Word2Vec embeddings, these learned embeddings are optimized for the specific task. A sentiment analysis model might learn embeddings where "great" and "wonderful" are close. A medical model might learn embeddings where "aspirin" and "ibuprofen" are close.
In a transformer (like GPT or BERT), the embedding layer is the very first component:
Tokenize the input text into token IDs: "The cat sat" → [464, 3797, 3332]
Look up each token's embedding vector from the embedding table
Add positional encoding so the model knows token order
Feed the resulting vectors into the transformer layers
The embedding table is learned end-to-end during training. By the time a model like GPT-4 finishes training on trillions of tokens, its embedding table contains incredibly rich representations of every token in its vocabulary.
How do we check if two word vectors are similar? We use cosine similarity -- the cosine of the angle between two vectors:
cos(a,b)=∥a∥∥b∥a⋅b
Cosine similarity ignores vector magnitude and focuses on direction. Two words pointing in the same direction in embedding space have cosine similarity near 1, regardless of how "long" their vectors are.
Try it: Explore Word Similarities in Embedding SpaceInteractive
Loading visualization...
Explore how words cluster in embedding space. Related words form tight groups -- animals cluster together, countries cluster together, verbs cluster together. Try searching for specific words and see which neighbors appear. Notice that the clusters capture meaning, not spelling -- "good" is near "great" and "excellent", not near "food" or "hood".
One-hot encoding fails for language -- it produces sparse, high-dimensional vectors with no notion of similarity between words
Word2Vec learns meaningful vectors from context -- CBOW predicts a word from its neighbors; Skip-gram predicts neighbors from a word; both learn that words in similar contexts get similar vectors
GloVe learns from co-occurrence statistics -- it factorizes a global word co-occurrence matrix into dense vectors, achieving similar quality to Word2Vec
Word arithmetic reveals learned structure -- relationships like gender, geography, and tense are encoded as consistent vector directions (king - man + woman ≈ queen)
Neural networks use learnable embedding layers -- instead of pre-trained vectors, modern models learn task-specific embeddings end-to-end via backpropagation
Cosine similarity measures semantic closeness -- the cosine of the angle between two embedding vectors tells you how similar their meanings are
Why does one-hot encoding fail as a word representation for NLP?
Next up: we already know how to represent words as vectors. But how does the model know which words in a sequence to pay attention to? That is the attention mechanism -- the idea that changed everything.