Embeddings: Categorical to Word2Vec

Don't worry if "embedding" sounds abstract — it is literally just a lookup table with one row per category, and gradient descent figures out what the rows should mean. Once you train one yourself you will see why this single trick replaced one-hot encoding everywhere it could.

#Why One-Hot Encoding Falls Apart

The problem with one-hot scales fast. Imagine you are building a YouTube recommender:

• 2 billion users. One-hot user ID = a 2,000,000,000-dim vector per row.
• 50 million videos. One-hot video ID = 50,000,000-dim vector.
• Every input row is mostly zeros. Storing a single batch eats more memory than fits on a GPU.
• The model cannot share statistical strength between similar users — every ID looks fully unrelated to every other ID.

Even at modest scales, one-hot wastes capacity. A categorical column with 35,000 unique zip codes adds 35,000 columns to a linear model. The model spends most of its parameter budget memorizing each ID rather than learning what they mean.

The embedding lookup is essentially a multiplication of the one-hot vector by the embedding matrix, but no real ML framework does it that way. Indexing a row out of a tensor is exactly equivalent and a thousand times faster.

#The Embedding Layer Is a Learned Lookup Table

import torch
import torch.nn as nn

# 10,000 vocab words, 64-dim vectors
embed = nn.Embedding(num_embeddings=10_000, embedding_dim=64)

ids = torch.tensor([42, 7, 999])     # batch of 3 ids
vectors = embed(ids)                  # shape (3, 64)
# vectors[0] is the row E[42], etc.

#Entity Embeddings for Tabular Data

The recipe

1. For each categorical column, choose an embedding dim. Rule of thumb: min(50, (V + 1) // 2) or fast.ai's min(600, round(1.6 * V**0.56)).
2. Stack the embeddings of every categorical for a row, concatenate with the numeric columns.
3. Feed the concatenated vector into an MLP.
4. Train end-to-end with the task loss.

Why it works: gradient descent learns a geometry where similar categories get similar vectors. For zip codes, "similar" turns out to mean "similar customer behaviour", not geographic proximity per se, but whatever predicts the target. The model discovers which categories should cluster.

For deep models, the embedding is almost always the right answer. Target encoding is a strong baseline for tree-based models (it leaks if not done with K-fold), but for a neural model that needs gradient flow through the whole pipeline, embeddings let the model discover its own zip-code geometry while training the prediction head.

#Word2Vec: Embeddings That Capture Meaning

Word embeddings are the most famous use case. The trick that made Word2Vec take off in 2013: train embeddings on a self-supervised task (predict surrounding words) using negative sampling for tractability.

#Skip-gram

(cat, the)
(cat, sat)
(cat, on)

#CBOW

Continuous Bag-of-Words is the mirror image: predict the centre word from its context. Average the context embeddings, then classify against the vocabulary. CBOW trains faster on small data; skip-gram works better on larger corpora and rare words.

#The Famous Vector Arithmetic

After training, the embeddings exhibit linear regularities that captured the world's attention:

vec("king") - vec("man") + vec("woman")  ≈  vec("queen")
vec("Paris") - vec("France") + vec("Italy")  ≈  vec("Rome")
vec("walking") - vec("walked") + vec("swam")  ≈  vec("swimming")

This was the moment when embeddings stopped being "a useful trick" and started being "language has geometry now". Every modern transformer's input layer is descended from this idea.

#GloVe: Combining Local + Global Statistics

GloVe's selling point: it sees the entire corpus statistics at once, not just a sliding window. In practice, Word2Vec and GloVe land within a few points of each other on most downstream evaluations, and the choice is mostly a vibes call, which library is easier to use.

#FastText: Sub-Word Embeddings for Open Vocabulary

The Achilles heel of Word2Vec and GloVe: they assign one vector per word and have no plan for words they have never seen. Misspellings, rare technical terms, morphologically rich languages (Finnish, Turkish), agglutinative compounds (German "Donaudampfschifffahrtsgesellschaft") all break.

• Out-of-vocabulary words get reasonable vectors. "Antidisestablishmentarianism" was never in the training data, but its sub-word n-grams were.
• Morphological relatives cluster naturally. "play", "plays", "playing", "played" all share most of their n-grams, so their vectors land close.

FastText is the right tool when your domain has open vocabulary — product titles, code identifiers, drug names, biological taxa, multilingual text. It loses to BERT on most modern NLP tasks, but it is much smaller, much faster, and runs on CPU.

#Modern Context: Embeddings Are Now the Input Layer of Transformers

#When to Use Which Embedding

#Visualising Embeddings

Once you have trained embeddings, project them to 2-D with t-SNE or UMAP and plot them. This is the iconic ML visualization where related concepts cluster:

• Word2Vec on Wikipedia: capitals cluster, animals cluster, professions cluster.
• Entity embeddings on Rossmann store IDs: stores in similar regions cluster, even though no region label was provided to the model.
• Image embeddings on ImageNet: dog breeds cluster together; vehicles cluster separately; tools form a third cluster.

The act of training to predict a target compresses semantic information into the embedding geometry, and projection makes it visible.

#Key Takeaways

• Embeddings are dense lookup tables. nn.Embedding(V, d) gives every category a learnable d-dim row, replacing V-dim sparse one-hot with a much smaller, similarity-aware representation
• Entity embeddings beat one-hot for high-cardinality tabular columns in deep models — fast.ai's Rossmann result and every modern recommender confirms this
• Word2Vec's skip-gram + negative sampling turned word embeddings into a viable production technique by reducing per-step cost from O(vocabulary) to O(K+1)
• Vector arithmetic emerges because training pushes co-occurring words close in space — king - man + woman ≈ queen is geometry that gradient descent built
• Modern transformers contain embedding layers as their first step, but the static Word2Vec/GloVe vectors are now mostly replaced by contextual embeddings produced by the full transformer

#Quick Check