"You shall know a word by the company it keeps." That 1957 quote by linguist J. R. Firth became the foundation of modern NLP in 2013, when Mikolov's Word2Vec paper showed that a tiny neural network could learn vector representations where king − man + woman ≈ queen falls out of the geometry. Today every recommendation system, search engine, and LLM is built on embeddings: dense vectors that put similar things near each other.
Learning Objectives
After this lesson, you will be able to:
Explain why one-hot encoding fails at scale and why a learned embedding lookup table compresses millions of categories into a few hundred dense, similarity-aware dimensions
Train your own categorical (entity) embeddings on a tabular dataset and beat one-hot + linear models with a small MLP that lets gradient descent discover the geometry of zip codes, product IDs, or user IDs
Implement Word2Vec skip-gram with negative sampling from scratch and observe the famous king − man + woman ≈ queen vector arithmetic emerge from raw text
Pick the right embedding strategy — entity, Word2Vec/GloVe, FastText, or pretrained contextual — based on cardinality, vocabulary openness, and the downstream task
Don't worry if "embedding" sounds abstract — it is literally just a lookup table with one row per category, and gradient descent figures out what the rows should mean. Once you train one yourself you will see why this single trick replaced one-hot encoding everywhere it could.
The problem with one-hot scales fast. Imagine you are building a YouTube recommender:
2 billion users. One-hot user ID = a 2,000,000,000-dim vector per row.
50 million videos. One-hot video ID = 50,000,000-dim vector.
Every input row is mostly zeros. Storing a single batch eats more memory than fits on a GPU.
The model cannot share statistical strength between similar users -- every ID looks fully unrelated to every other ID.
Even at modest scales, one-hot wastes capacity. A categorical column with 35,000 unique zip codes adds 35,000 columns to a linear model. The model spends most of its parameter budget memorizing each ID rather than learning what they mean.
xonehot∈{0,1}Vvsei=E[i,:]∈Rd,d≪V
The embedding lookup is essentially a multiplication of the one-hot vector by the embedding matrix -- but no real ML framework does it that way. Indexing a row out of a tensor is exactly equivalent and a thousand times faster.
In PyTorch, an embedding layer is just nn.Embedding(num_embeddings=V, embedding_dim=d). Internally, this allocates a (V, d) parameter tensor E. Forward pass on input id i returns E[i]. Backward pass routes gradients only to row E[i].
pythonreference · read-only
1
2
3
4
5
6
7
8
9
import torch
import torch.nn as nn
# 10,000 vocab words, 64-dim vectors
embed = nn.Embedding(num_embeddings=10_000, embedding_dim=64)
ids = torch.tensor([42, 7, 999]) # batch of 3 ids
vectors = embed(ids) # shape (3, 64)
# vectors[0] is the row E[42], etc.
The trainable parameter count is V * d. For 10K vocab × 64 dim = 640K params -- a small MLP layer. For 1M vocab × 256 dim = 256M params -- a serious chunk of a model, but still fits on one GPU. (Modern recommenders that need 100M+ vocabularies use sharded embedding tables across GPUs; we will hit that in the Distributed Training lesson.)
The first place embeddings escape NLP is tabular ML. Cheng Guo's 2016 insight: anywhere you have a high-cardinality categorical, swap one-hot for nn.Embedding.
The recipe
For each categorical column, choose an embedding dim. Rule of thumb: min(50, (V + 1) // 2) or fast.ai's min(600, round(1.6 * V**0.56)).
Stack the embeddings of every categorical for a row, concatenate with the numeric columns.
Feed the concatenated vector into an MLP.
Train end-to-end with the task loss.
Why it works: gradient descent learns a geometry where similar categories get similar vectors. For zip codes, "similar" turns out to mean "similar customer behaviour" -- not geographic proximity per se, but whatever predicts the target. The model discovers which categories should cluster.
ei=E[i,:]whereE∈RV×dandd=min(50,⌈(V+1)/2⌉)
What Do You Think?
You are predicting credit risk and have a zip_code column with 35,000 unique values. You also have 50 numeric features. Which encoding do you reach for first?
For deep models, the embedding is almost always the right answer. Target encoding is a strong baseline for tree-based models (it leaks if not done with K-fold), but for a neural model that needs gradient flow through the whole pipeline, embeddings let the model discover its own zip-code geometry while training the prediction head.
Word embeddings are the most famous use case. The trick that made Word2Vec take off in 2013: train embeddings on a self-supervised task (predict surrounding words) using negative sampling for tractability.
Given a centre word, predict the words around it. For "the cat sat on the mat", with centre word cat and window size 2, the (target, context) pairs are:
(cat, the)
(cat, sat)
(cat, on)
The model learns two embedding matrices: U for centre words and V for context words (some implementations share them). The probability of context word $w_O$ given centre word $w_I$ is a softmax:
P(wO∣wI)=∑w∈Vexp(vw⋅uwI)exp(vwO⋅uwI)
The full softmax sums over the entire vocabulary -- 50K to 1M words -- at every step. Mikolov's key efficiency trick was negative sampling: replace the full softmax with a binary classification "is this a real context word, or a randomly sampled negative?". For each positive (centre, context) pair, sample 5-20 random negatives (words not actually in the context) and train a binary cross-entropy.
Continuous Bag-of-Words is the mirror image: predict the centre word from its context. Average the context embeddings, then classify against the vocabulary. CBOW trains faster on small data; skip-gram works better on larger corpora and rare words.
The reason these emerge: skip-gram pushes words appearing in similar contexts to have similar vectors, and "king" and "queen" appear in mostly-similar contexts plus a gender-shifted dimension. The vector difference king - queen ends up encoding gender, and gradient descent makes that gender direction approximately consistent across pairs.
This was the moment when embeddings stopped being "a useful trick" and started being "language has geometry now". Every modern transformer's input layer is descended from this idea.
GloVe (Pennington et al. 2014) reframes embeddings as matrix factorization. Build a global word-co-occurrence count matrix X where X_ij = how often word j appears in the context of word i. Then learn vectors w_i and \tilde{w}_j such that:
J=i,j=1∑Vf(Xij)(wi⊤w~j+bi+b~j−logXij)2
GloVe's selling point: it sees the entire corpus statistics at once, not just a sliding window. In practice, Word2Vec and GloVe land within a few points of each other on most downstream evaluations, and the choice is mostly a vibes call -- which library is easier to use.
#FastText: Sub-Word Embeddings for Open Vocabulary
The Achilles heel of Word2Vec and GloVe: they assign one vector per word and have no plan for words they have never seen. Misspellings, rare technical terms, morphologically rich languages (Finnish, Turkish), agglutinative compounds (German "Donaudampfschifffahrtsgesellschaft") all break.
FastText (Bojanowski et al. 2017) embeds character n-grams instead. The vector for a word is the sum of its character-n-gram vectors. The word unhappy is built from <un, unh, nha, hap, app, ppy, py> (with < and > as word boundaries). Two big payoffs:
Out-of-vocabulary words get reasonable vectors. "Antidisestablishmentarianism" was never in the training data, but its sub-word n-grams were.
Morphological relatives cluster naturally. "play", "plays", "playing", "played" all share most of their n-grams, so their vectors land close.
FastText is the right tool when your domain has open vocabulary -- product titles, code identifiers, drug names, biological taxa, multilingual text. It loses to BERT on most modern NLP tasks, but it is much smaller, much faster, and runs on CPU.
#Modern Context: Embeddings Are Now the Input Layer of Transformers
Word2Vec and GloVe gave each word one vector regardless of context. "Bank" had a single vector that averaged "river bank" and "savings bank", which is obviously wrong.
Contextual embeddings -- ELMo (2018), BERT (2018), GPT (2018) -- give each word a vector per occurrence. The embedding for "bank" depends on the words around it. This is why every modern NLP model uses transformer-based embeddings as the input layer rather than a static Word2Vec lookup. We will cover the transformer architecture and contextual embeddings in track-06 NLP & Transformers.
But the conceptual descendant runs inside every transformer: the input embedding layer is still nn.Embedding(vocab_size, hidden_dim). Word2Vec did not die -- it became the first layer of every LLM you have ever used.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
Tests · Run the script. The top-1 closest word to king - man + woman should usually be queen (or sometimes princess). With more data and dimensions, the vector arithmetic gets sharper.
Once you have trained embeddings, project them to 2-D with t-SNE or UMAP and plot them. This is the iconic ML visualization where related concepts cluster:
Word2Vec on Wikipedia: capitals cluster, animals cluster, professions cluster.
Entity embeddings on Rossmann store IDs: stores in similar regions cluster, even though no region label was provided to the model.
Image embeddings on ImageNet: dog breeds cluster together; vehicles cluster separately; tools form a third cluster.
The act of training to predict a target compresses semantic information into the embedding geometry, and projection makes it visible.
Embeddings are dense lookup tables. nn.Embedding(V, d) gives every category a learnable d-dim row, replacing V-dim sparse one-hot with a much smaller, similarity-aware representation
Entity embeddings beat one-hot for high-cardinality tabular columns in deep models — fast.ai's Rossmann result and every modern recommender confirms this
Word2Vec's skip-gram + negative sampling turned word embeddings into a viable production technique by reducing per-step cost from O(vocabulary) to O(K+1)
Vector arithmetic emerges because training pushes co-occurring words close in space — king - man + woman ≈ queen is geometry that gradient descent built
Modern transformers contain embedding layers as their first step, but the static Word2Vec/GloVe vectors are now mostly replaced by contextual embeddings produced by the full transformer
Why does negative sampling speed up Word2Vec training?
Embeddings turn discrete IDs into continuous geometry the network can reason over. Next up: Multi-Head Attention — how a transformer lets every token look at every other token, so those static vectors become context-aware representations.