"Encoding 'Monday=1, Tuesday=2' looks innocent — until your linear model decides Sunday is seven times bigger than Monday. Encoding 'zip code' as an integer looks innocent — until your model learns that ZIP 90210 is 89,999 units 'larger' than ZIP 00211. The encoder you pick literally rewrites the geometry of your data. Get it wrong and the smartest model in the world will fit nonsense beautifully."
Learning Objectives
After this lesson, you will be able to:
Tell the difference between nominal categories (red, blue, green), ordinal categories (small, medium, large), and cyclic categories (Monday-Sunday) — because the wrong encoder for the wrong type quietly teaches your model nonsense
Pick the right encoder by cardinality — one-hot for fewer than ~5 unique values, target encoding for 5-1000, and the hashing trick or learned embeddings above that — using a clear decision rubric instead of guessing
Apply target encoding safely with K-fold or leave-one-out smoothing so the encoder never peeks at the row it's encoding, which is the most common silent leakage source in tabular ML
Use the hashing trick to encode millions of categories with a fixed memory footprint, and use entity embeddings to let neural networks learn category meaning end-to-end the way fast.ai showed in the Rossmann challenge
Why fast.ai's entity embeddings beat XGBoost on the Rossmann store sales challenge -- Cheng Guo's 2016 paper "Entity Embeddings of Categorical Variables" showed that letting a neural network learn a dense vector per store_id (instead of one-hot) lifted Rossmann sales prediction from rank 33 to rank 3 on the leaderboard; the same trick now powers every modern tabular deep learning library
Build this --> Take a CSV of Spotify song genres or Amazon product categories, encode the same column three different ways (one-hot, target, hash) on the same downstream model, and compare validation RMSE -- you will feel the cardinality decision rubric in your bones after one experiment
Don't worry if "encoder" sounds intimidating — it just means "turn this string into a number a model can do math on." That's the whole job. The art is doing it without lying to your model.
Before you pick an encoder, classify what you're holding. The three types demand three different encoders.
The single most common encoding bug is treating a nominal variable as ordinal because the strings happen to sort alphabetically. We will hammer on this throughout.
The drop_first trick. If you know that exactly one bulb is always on, you can drop one column without losing information -- the dropped category is implied when all other columns are 0. This avoids the dummy variable trap in linear models (perfect multicollinearity inflates standard errors), but it does NOT matter for tree-based models, which split on individual columns regardless.
Pros: No false ordering. Works for any nominal category. Sparse-matrix-friendly.
Cons: Cardinality K becomes K columns. Memory blows up at K > 50. New categories at inference time become all-zero rows -- a silent failure unless you set handle_unknown='ignore'.
#Ordinal & Label Encoding: Numbers, In Order (or Not)
Ordinal encoding assigns integers to categories with a meaningful ordering you provide.S=0, M=1, L=2, XL=3 is ordinal -- you told the encoder the order, and the model can use the fact that L > M.
Label encoding is the same machinery but with no ordering guarantee -- sklearn's LabelEncoder just sorts categories alphabetically. Use this only for the target variable in a classification problem, never for input features unless you genuinely have an ordinal column.
ordinal(xi)=rank(xi)∈{0,1,…,K−1}
Pros: One column. Memory cheap. Trees handle it natively (a split at "size <= 1.5" is just splitting between M and L).
Cons: Linear and distance-based models (linear regression, k-NN, SVM) interpret the integer as a real number, so ordinal-encoding a nominal column teaches them lies.
#Target Encoding: The Average Goal of the Category
target(c)=nc+mnc⋅yˉc+m⋅yˉglobal
The smoothing parameter m is the unsung hero. Without it, a category that appears only once gets encoded as that single row's exact target -- which is just memorization with extra steps. With m=20, that singleton category gets pulled mostly toward the global mean and the model can no longer abuse it.
The K-fold protocol (production-grade target encoding):
for each fold f in K folds:
- split data into fold_f and rest
- compute category means on `rest`
- apply those means to `fold_f`
finally: at inference, use category means computed from the FULL training set
This is the only safe way to target-encode without leaking. The sklearn TargetEncoder (added in version 1.3, October 2023) implements this internally with cv=5 by default -- before that, every Kaggle notebook rolled its own.
What Do You Think?
You have a zip_code column with 35,000 unique values, and you're training a linear regression to predict household income. Which encoder should you reach for first?
K-fold target encoding is the right call. One-hot creates 35,000 columns of mostly noise. Label-encoding teaches the model that zip 90211 is 35,000 units larger than zip 00001. Dropping it loses one of the most predictive signals you have. K-fold target encoding compresses the same information into one well-behaved column -- and the K-fold split is what makes it leakage-free.
Replace each category with how often it appears in the dataset. country = US becomes 0.42 if 42% of rows are US; country = LU (Luxembourg) becomes 0.0001.
This is a niche but useful trick. It captures the signal that "rare categories behave differently" without leaking the target. Useful as a secondary feature alongside another encoder, or when you have no target (unsupervised). Not powerful enough to stand alone for high-stakes problems.
#The Hashing Trick: Fixed Memory For Unbounded Categories
When you have millions of categories -- pin IDs, product SKUs, ad creatives, user IDs -- one-hot is impossible (memory) and target encoding is fragile (most categories appear only once or twice). The hashing trick gives you a fixed output dimension regardless of how many categories exist now or might exist tomorrow.
h(xi)=hash(xi)modNthen onehot(h(xi))∈{0,1}N
Why collisions are tolerable. With N = 2^20 = ~1M buckets and 350M categories, the expected number of categories per bucket is ~350. But almost every bucket contains a random mixture of categories; the model can pick up the signal of the truly informative categories and average over the noise from the rest. This is the same logic that makes Bloom filters work.
Pros: Unbounded cardinality, fixed memory, no fit step required (hashing is stateless). Streaming-friendly. New unseen categories at inference work automatically -- they just hash into a bucket like everything else.
Cons: No interpretability (you can't ask "what's in bucket 314,159?"). Tuning N is empirical. Inverse hashing is impossible -- you can't recover the original category from the hash, which complicates debugging.
#Entity Embeddings: Let The Network Learn The Meaning
The deep-learning answer to high-cardinality categoricals: instead of one-hot or hash, learn a small dense vector per category and let backprop tune it.
embed(c)=E[c]∈Rdwhere E∈RK×d
This is exactly word2vec's mechanism applied to non-text categoricals. fast.ai's TabularModel uses one embedding layer per categorical column, then concatenates the resulting vectors with the numeric features and feeds the whole thing into an MLP. The embeddings are tuned end-to-end with the rest of the network.
Pros: Learns category similarity -- after training, similar stores or zip codes have similar vectors, so the model generalizes to rare categories better than target encoding. Outperforms one-hot at high cardinality on tabular deep learning benchmarks.
Cons: Requires a neural network (XGBoost can't do it natively, though TabNet and the like try). Needs enough rows per category for the embeddings to learn -- categories appearing only 1-2 times stay near their random initialization.
Use this as your default. Override it only with a measured experiment.
Compare encoders on the same dataset and watch the column count and downstream model performanceInteractive
Loading visualization...
Cardinality K
First-choice encoder
Why
K < 5
One-hot
Cheap, interpretable, no leakage risk
5 ≤ K ≤ 50
One-hot for trees, target for linear models
Either works; trees prefer one-hot, linear/distance models love target
50 < K ≤ 1,000
Target encoding (K-fold)
One-hot would be 1000 sparse columns; target captures signal in one
1,000 < K ≤ 100,000
Target encoding or hashing
Target if memory permits and rows-per-category is healthy; hash if streaming or memory-constrained
K > 100,000
Hashing or embeddings
One-hot is impossible; target encoding becomes unstable; hashing for non-DL, embeddings for DL
Cross-cutting modifiers. Tree models (Random Forest, XGBoost, LightGBM, CatBoost) handle integer-encoded categoricals well -- LightGBM and CatBoost have native categorical support that beats most hand-rolled encoders. Linear models (LinearRegression, LogisticRegression, SVM) and distance models (k-NN, k-means) demand careful encoding because they treat numbers as real-valued. When in doubt, build a small benchmark that swaps the encoder and report validation RMSE -- it takes 10 minutes and beats every rule of thumb in print.
What happens when production sees a category your encoder has never seen during training?
One-hot: with handle_unknown='ignore', returns an all-zero vector. Without that flag, throws an exception. Always set handle_unknown='ignore' in production code.
Ordinal: sklearn returns -1 by default. Watch out -- a downstream linear model now sees a negative number from a column it thought was non-negative.
Target encoding: sklearn's TargetEncoder falls back to the global mean. This is a sensible default and almost always what you want.
Hashing: unseen categories hash into some bucket like everything else. No special handling needed -- this is hashing's superpower.
Embeddings: the standard trick is to reserve embedding row 0 for an <UNK> token and route any unseen category to it. PyTorch's nn.Embedding makes this a one-line setup.
Tests · Verify all three encoders produce sensible MSE (target should be lowest because the synthetic data has a strong city-level signal). Verify hashing improves with larger N.
Pick the encoder by category type AND cardinality. Nominal needs one-hot or hashing, ordinal accepts integer encoding, cyclic needs sin/cos; cardinality below ~50 favors one-hot, above that favors target encoding, hashing, or embeddings
Target encoding without K-fold is leakage with extra steps. Always cross-fit the means on training folds and use full-training-set means at inference; sklearn's TargetEncoder (1.3+) does this for you with cv=5 by default
Smoothing m saves rare categories. Without it, a category appearing once gets encoded as that single row's exact target, which is memorization rather than learning; m=10-30 is a sensible default
Hashing trades collisions for unbounded cardinality. At 2^20 buckets and 350M categories you collide constantly, but the model averages over collisions and you keep a fixed memory footprint, which is the only thing that makes web-scale recommendation systems possible
Entity embeddings beat target encoding when you have a neural network and enough data. Cheng Guo's Rossmann result (rank 33 → rank 3 with embeddings as the only change) is the canonical demonstration; tabular DL libraries like fast.ai's bake this in by default
You're training a linear regression with a `country` column that has 195 unique values. Which encoder should you reach for first?
The one rule that survives every encoder choice and every downstream model: fit on training rows, transform validation and test. Everything else is just picking the right tool for the cardinality.
Encoders turn names into numbers; encoding well turns names into useful numbers without lying. Next up: Feature Engineering — where you take your now-numeric inputs and combine, scale, and reshape them into the features your model actually wants to see.