RugvAI Labs

Feature Engineering: The 80% of ML

Walk the hallway of any Kaggle grandmaster, any fraud team at Stripe, any churn analyst at Spotify, and ask them what actually moved the needle on their last big win. Almost nobody says "I switched from XGBoost to LightGBM." Almost everybody says some version of: "I built a feature that captured X." The unglamorous truth of applied ML is that the model is usually a commodity — sklearn ships dozens of them, all roughly equivalent on clean data — and the differentiator is the features you feed it. Andrew Ng calls this "data-centric AI." Pedro Domingos calls it "the unreasonable effectiveness of features." Practitioners just call it the job.

This lesson is the deep dive on that job. You will learn how to scale, transform, encode, impute, generate, and not leak — across numerical, categorical, text, and time features — and you will write a real preprocessing pipeline that production teams ship every day.

Learning Objectives

After this lesson, you will be able to:

Choose the right scaler (StandardScaler, MinMaxScaler, RobustScaler, Normalizer) for a given feature distribution and downstream model
Transform skewed and zero-heavy features with log, sqrt, Box-Cox, and Yeo-Johnson — and know which one to reach for first
Encode categorical variables intelligently — one-hot, ordinal, target encoding with smoothing, hashing trick — based on cardinality and leakage risk
Engineer time, text, and missing-data features that capture the structure flat tables hide
Build a leak-proof sklearn Pipeline + ColumnTransformer that fits only on training data and is safe to deploy

Prerequisites

Before You Start

Review

Why

Why This Matters

Kaggle winners rarely have novel models. They have novel features. The Mercari Price Suggestion winners used the same gradient boosting library as the 200th-place team; what put them on the podium was target encoding with leave-one-out cross-validation and engineered text features from product descriptions
Fraud detection at scale runs on engineered features. Stripe's Radar, Sift, and every major card network do not feed raw transaction rows into a model. They feed velocity features (transactions in the last 5 minutes, last hour, last day), graph features (how many devices has this card seen?), and rolling aggregates that only exist because someone wrote the code to compute them
Churn, LTV, ranking, ad CTR. Every tabular ML system in industry is a contest of features, not algorithms. Production teams spend roughly 80% of their cycles on data + features and 20% on modeling. Get this lesson right and you have just gotten promoted
Build this --> Take the Titanic dataset, build a Pipeline with ColumnTransformer that scales numerical features, target-encodes high-cardinality categoricals, and adds a "missingness" flag. Beat the default Random Forest baseline by 4+ accuracy points without touching the model

The reason this lesson is 40 minutes long instead of 10 is that "feature engineering" is not one trick — it is a discipline. The principles are simple, but the failure modes are everywhere, and most of them silently inflate your cross-validation score without you noticing. By the end you will have a mental checklist that survives contact with messy real data.

#Why Features Matter More Than Models

Intuition

Build Your Intuition

Features are how you tell the model what matters. A model is a function approximator — it learns the mapping from inputs to outputs that you give it. If your input is a raw timestamp 1715600000, the model has to discover from scratch that 6 PM on Friday is different from 6 AM on Tuesday. If your input is [hour=18, day_of_week=4, is_weekend=False], you have done that work for it.

Every feature you engineer is a chunk of domain knowledge you have handed the model on a silver platter. You are no longer asking it to figure out everything from raw bytes. You are asking it to interpolate between the shortcuts you have already drawn.

Deep learning weakens this idea — neural networks learn representations from raw pixels, raw text, raw audio. But for tabular data, where rows are heterogeneous and signals are sparse, hand-engineered features still beat learned representations more often than not. This is why XGBoost and LightGBM still dominate Kaggle in 2026.

A useful mental model: a learning algorithm has a fixed hypothesis space — the set of functions it can express. Linear regression's hypothesis space is "weighted sums of inputs." A decision tree's is "axis-aligned splits." When you engineer a feature, you are moving the data into a region of that hypothesis space where the model can fit it well. You are not making the model smarter; you are making the problem easier.

This is why the same algorithm can win or lose by 20 points depending on who is driving. Two data scientists, one XGBoost, two completely different feature sets — two completely different leaderboard finishes.

#Numerical Features: Scaling

Most ML algorithms have an implicit assumption about how features are scaled. Get the scaling wrong and the model misbehaves silently — coefficients explode, gradients vanish, regularization punishes the wrong things, distance metrics get bullied by the feature with the biggest units.

There are four scalers you should know cold.

#StandardScaler (z-score)

The most common: subtract the mean, divide by the standard deviation. Each transformed feature has mean 0 and standard deviation 1.

z = \frac{x - \mu}{\sigma}

Use StandardScaler when: features are roughly symmetric, you are using a linear model (linear/logistic regression, ridge, lasso), an SVM with RBF/linear kernel, k-means, k-NN, PCA, or a neural network. Basically the default for anything that has a notion of distance or that regularizes weights.

#MinMaxScaler

Squeeze each feature into a fixed range, usually [0, 1].

x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}

Use MinMaxScaler when: you specifically need values in [0, 1] (image pixels, neural network inputs that feed sigmoids, anything bounded). The downside: it is extremely sensitive to outliers. One outlier pushes x_max way out, and 99% of your data ends up squeezed into a sliver near zero.

#RobustScaler

Use the median and IQR instead of mean and std. Outlier-resistant.

x' = \frac{x - \text{median}(x)}{\text{IQR}(x)}

Use RobustScaler when: you have known outliers you cannot or do not want to remove (fraud amounts, salary, real estate prices in a city with mansions). The median ignores the tails; the mean does not.

#Normalizer

Different from the others — it normalizes each row (sample) to unit norm. Most often used for text vectors (TF-IDF) and cosine-similarity workloads.

\mathbf{x}' = \frac{\mathbf{x}}{\|\mathbf{x}\|_2}

Use Normalizer when: you care about direction, not magnitude. TF-IDF + cosine similarity is the canonical case. Document length should not determine similarity.

What Do You Think?

You are building a Random Forest or gradient boosting model. Do you need to scale your numerical features?

The rule of thumb worth memorizing: anything that measures distance or regularizes weights needs scaling. Trees do not.

#Numerical Features: Skew Transformations

Real-world numerical data is rarely Gaussian. Income is right-skewed (a few people make millions, most make tens of thousands). House prices are right-skewed. Click counts, session durations, file sizes — all right-skewed. A linear model trying to fit a feature where 99% of values are between 1 and 100, but 1% are between 10,000 and 1,000,000, gets pulled around by the long tail.

Three transformations solve this:

#Log Transform

x' = log(x + 1) (the +1 avoids log(0)).

The classic. Compresses the long right tail and pulls extreme values back toward the bulk. After log(1 + x), income distributions are roughly Gaussian. Click counts become roughly Gaussian. Anything with multiplicative structure (where doubling matters more than adding 100) benefits.

Requires strictly positive inputs.

#Square Root

x' = sqrt(x). Milder than log. Use for count data with low values (number of children, number of doctor visits per year) where log would over-compress.

#Box-Cox

A family of power transforms with a single parameter λ that finds the optimal power for normality:

x'(\lambda) = \begin{cases} \dfrac{x^{\lambda} - 1}{\lambda} & \lambda \ne 0 \\ \ln(x) & \lambda = 0 \end{cases}

The catch: Box-Cox requires strictly positive values. Has any zero or negative number? Box-Cox crashes.

#Yeo-Johnson

Box-Cox's better-behaved cousin. Same idea, but defined piecewise so it handles zero and negative numbers natively. In practice, Yeo-Johnson is the safe default — you almost never lose anything vs. Box-Cox, and you do not have to filter out zeros or shift your data.

Quick check

You have a 'net_profit' feature that ranges from -$50,000 to +$2,000,000 (some companies lose money). You want a power transform to make it roughly Gaussian. Which do you reach for?

#Numerical Features: Binning

Sometimes the right move is to throw away precision. Convert a numerical feature into discrete buckets.

Equal-width binning: Divide the range into k equal-size intervals. Simple, but unbalanced if the distribution is skewed (most of your data ends up in one bin).
Equal-frequency (quantile) binning: Each bin has roughly the same number of samples. More robust to skew.
Supervised binning: Use the target to find the bins. Decision-tree-based discretization (one tree per feature, fit on (x, y), use the leaves as bins) captures non-linear relationships a linear model could not.

Why bin? Three reasons:

Captures non-linearity in linear models — age binned into [0-18, 18-25, 25-40, 40-65, 65+] gives a linear model the ability to express "older customers spend more, but seniors spend less than middle-aged" without polynomial features.
Robust to outliers — top bin is just "the top bin." Doesn't matter if the value is $10M or $1B.
Interpretability — "income bracket" is human-readable. "log of income normalized" is not.

The cost: you throw away ordering precision. A bin says "between 40 and 65," but loses the distinction between 41 and 64.

#Numerical Features: Interactions

Models like linear regression cannot discover that price is roughly sqft * price_per_sqft. You have to hand it the product. PolynomialFeatures(degree=2) in sklearn creates all pairwise products and squares of your features. For n features, you get O(n²) new features — explodes quickly.

In practice, you do not blindly polynomial-expand everything. You either:

Use domain knowledge to construct specific interactions (bmi = weight / height², revenue_per_user = revenue / dau).
Let a tree model discover them implicitly (trees can express any interaction by chaining splits, which is why they often need less feature engineering than linear models).

#Categorical Features

Categorical features are where most beginners get stuck. They cannot be fed to most models directly — you have to encode them as numbers. How you do that matters a lot.

#One-Hot Encoding

The standard for low-to-medium cardinality (under ~50 unique values). Each category becomes its own 0/1 column.

pythonrunnable cell

from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="ignore",   # don't crash on new categories at predict time
    sparse_output=False,
    drop="first",              # drop one column to avoid the dummy variable trap
)
X_encoded = ohe.fit_transform(X_categorical)

The dummy variable trap and drop="first": if you have a feature color in {red, green, blue} and you create three columns is_red, is_green, is_blue, they sum to exactly 1 for every row. That means one is a linear combination of the other two and the bias term — perfect multicollinearity. For linear regression with no regularization, the matrix X^T X becomes singular and the normal equation fails. The fix: drop="first" to drop one category and make it the implicit baseline (encoded as all-zeros).

For tree models and regularized linear models, drop="first" is unnecessary (the regularizer breaks the degeneracy), but it does not hurt.

#Ordinal Encoding

Map categories to integers 0, 1, 2, .... Only valid when there is a real, meaningful order to the categories.

pythonrunnable cell

from sklearn.preprocessing import OrdinalEncoder

# CORRECT use: education has a natural order
size_encoder = OrdinalEncoder(categories=[
    ["high_school", "bachelors", "masters", "phd"]
])

# WRONG use: cities have no order, but this would force one
# city_encoder = OrdinalEncoder()  # would map alphabetically — meaningless

Quick check

You have a 'city' feature with values {New York, Los Angeles, Chicago, Houston}. You ordinal-encode it as {NYC=0, LA=1, CHI=2, HOU=3} and feed it to a linear regression for house price. What goes wrong?

#Target / Mean Encoding (with smoothing and cross-fold)

The killer technique for high-cardinality categoricals (hundreds or thousands of unique values). Instead of creating one column per category, replace each category with the mean of the target for that category.

zipcode  avg_house_price
10001    1,200,000
60601      650,000
77002      420,000
...

A feature with 30,000 zip codes becomes a single highly informative numerical column. One-hot would have produced 30,000 mostly-zero columns and overfit catastrophically.

The catch: target encoding leaks if you do not smooth and cross-validate it. Two bugs to know:

1. Without smoothing: A category with 1 sample gets that sample's exact target value, which is pure noise. Smooth toward the global mean using a Bayesian shrinkage:

\hat{y}_{\text{cat}} = \frac{n_{\text{cat}} \cdot \bar{y}_{\text{cat}} + m \cdot \bar{y}_{\text{global}}}{n_{\text{cat}} + m}

2. Without cross-fold encoding: If you compute the mean target per category on the entire training set and feed it as a feature, the model sees the target of each row baked into its input. The training R² will be artificially high; cross-validation will look great; production will be a disaster. The fix is out-of-fold target encoding: for each fold, compute category means using only the other folds. sklearn has TargetEncoder (added in 1.3) that does this for you, or use category_encoders.TargetEncoder with cv set.

#Frequency Encoding

Simpler cousin: replace each category with how often it appears.

pythonrunnable cell

freq_map = X_train["category"].value_counts(normalize=True).to_dict()
X_train["category_freq"] = X_train["category"].map(freq_map)
X_test["category_freq"]  = X_test["category"].map(freq_map).fillna(0)

Cheap, no leakage (counts do not use the target), often surprisingly useful — rare categories are often qualitatively different from common ones, and a single frequency column captures that.

#Hashing Trick

For genuinely massive cardinality — millions or billions of unique categories (URLs, IPs, ad creative IDs). One-hot is infeasible; even target encoding requires storing a lookup table per category. The hashing trick hashes each category to one of k fixed buckets:

pythonrunnable cell

from sklearn.feature_extraction import FeatureHasher

hasher = FeatureHasher(n_features=2**18, input_type="string")
X_hashed = hasher.transform([[s] for s in X_train["url"]])

Collisions happen (two different categories land in the same bucket), but with k = 2^18 = 262,144 buckets and most categories rare, collisions add a tolerable amount of noise. This is how online ad systems and real-time recommendation engines handle billion-category sparse data.

What Do You Think?

Your 'user_id' feature has 50,000 unique values, with a long-tail distribution (a few users appear thousands of times, most appear once or twice). Which encoder should you reach for first?

#Embeddings (Forward Reference)

The deep-learning answer to high-cardinality categoricals: instead of hashing or target-encoding, learn a dense vector per category jointly with the rest of the model. An nn.Embedding(num_users=50_000, dim=16) maps each user to a 16-dimensional vector that gets optimized end-to-end. This is how recommendation systems at YouTube, Netflix, TikTok, and Spotify handle user/item IDs. We come back to this in the deep learning track — for now, the takeaway is: when you outgrow target encoding, you do not start one-hot encoding harder. You learn embeddings.

#Text Features (Quick Coverage)

Text is its own world (full lesson in the NLP track), but for tabular ML with a text column, two techniques cover 90% of cases:

TF-IDF: Each document becomes a sparse vector where each dimension is a word, weighted by how informative that word is. Words that appear in every document (the, and, of) get low weights; rare-but-not-too-rare words get high weights. TfidfVectorizer in sklearn.

Character n-grams: Instead of words, use overlapping n-character substrings. Robust to misspellings, captures word fragments, works for languages without clean word boundaries. TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5)).

For deep learning text (BERT, sentence transformers), see the NLP track. For a free-text column inside a tabular pipeline, TF-IDF + char n-grams + truncated SVD (to reduce dimensionality) is the boring-but-strong workflow.

#Date/Time Features

A timestamp is the most under-engineered feature in most pipelines. Raw Unix time is almost useless to a linear model — 1715600000 does not mean anything. Expand it into components:

pythonrunnable cell

df["hour"]        = df["ts"].dt.hour
df["day_of_week"] = df["ts"].dt.dayofweek   # 0 = Monday
df["day"]         = df["ts"].dt.day
df["month"]       = df["ts"].dt.month
df["quarter"]     = df["ts"].dt.quarter
df["year"]        = df["ts"].dt.year
df["is_weekend"]  = df["day_of_week"].isin([5, 6]).astype(int)
df["is_holiday"]  = df["ts"].dt.date.isin(US_HOLIDAYS).astype(int)
df["days_since_signup"] = (df["ts"] - df["signup_ts"]).dt.days

For periodic features (hour of day, day of year, day of week) there is one more trick: use cyclical encoding. Hour 23 and hour 0 are next to each other on a clock, but a linear model sees them 23 units apart. Fix this with sine/cosine pair:

\text{hour}_{\sin} = \sin\!\left(\frac{2\pi \cdot \text{hour}}{24}\right), \quad \text{hour}_{\cos} = \cos\!\left(\frac{2\pi \cdot \text{hour}}{24}\right)

The pair (sin, cos) is needed — using only one collapses two different hours to the same value (hour 6 and hour 18 both have sin(2π·6/24) = sin(2π·18/24) = 1 and -1 respectively, but cos distinguishes them).

For tree models, cyclical encoding is less important — trees can split on hour ≤ 0 and hour ≥ 22 separately and recover the cyclical structure. For linear models, neural networks, and SVMs, cyclical encoding is the difference between a model that thinks midnight is far from 1 AM and one that doesn't.

#Missing Data

Real data has holes. Sometimes a value is missing at random (a sensor dropped a reading); sometimes the fact that it is missing is itself a signal (a customer who didn't fill in their income is qualitatively different from one who did).

#Imputation Strategies

Mean / median / mode imputation: Fast, default, works. Median is more robust to outliers; mode is the natural choice for categoricals.
KNN imputation: For each missing value, find the k rows most similar on the other (non-missing) features and average their values. More accurate, much slower, requires scaling first.
Iterative imputation (MICE): Treat each column with missing values as a regression problem with the other columns as predictors. Iterate until stable. IterativeImputer in sklearn. Most accurate, also the most expensive.

#The "Missingness as a Feature" Trick

Often overlooked: before imputing, add a binary is_missing column for each feature with missing values:

pythonrunnable cell

for col in numerical_cols_with_missing:
    df[f"{col}_was_missing"] = df[col].isna().astype(int)
    df[col] = df[col].fillna(df[col].median())

Why this helps: missingness is rarely random. "Didn't fill in income on a loan application" is a strong signal of risk. "Skipped the optional survey question" correlates with churn. The model gets two pieces of information from one column: the imputed value and the fact that the original was missing.

#When to Leave NaN Alone

XGBoost, LightGBM, CatBoost, and modern sklearn HistGradientBoosting all handle NaN natively — they learn a default direction at each split for missing values. For these models, imputing can hurt slightly by destroying the signal in the NaN pattern. Pass NaNs straight through and let the model decide.

For linear models, k-NN, SVMs, neural networks — you must impute. NaN inputs break the math.

#Feature Selection

Once you have generated 200 features, which ones do you keep? Three families of methods.

#Filter Methods

Score each feature independently against the target, keep the top k. Fast, model-agnostic.

Variance threshold: Drop features with near-zero variance (constant or near-constant columns are useless).
Chi-square / mutual information: Statistical association between feature and target. SelectKBest(score_func=mutual_info_classif, k=50).

Cheap and easy. Weakness: ignores feature interactions — a feature can be useless alone but powerful in combination, and filter methods will throw it away.

#Wrapper Methods

Search over subsets of features, evaluating each subset by training a model and measuring CV score.

Recursive Feature Elimination (RFE): Start with all features, train a model, drop the weakest, retrain, repeat until you reach the desired count.
Sequential Forward / Backward Selection: Greedy search adding or removing one feature at a time.

Better than filter methods because they respect interactions. Much slower — you train a model for every subset evaluated.

#Embedded Methods

The model itself does the selection during training. Best of both worlds.

L1 regularization (Lasso): As you saw in the regularization lesson, Lasso drives weights of unimportant features to exactly zero. Increase alpha until you have the count you want.
Tree feature importance: Random Forest and gradient boosting report feature_importances_ — how much each feature reduced impurity across all trees. Top-k by importance.
Permutation importance: For each feature, shuffle its values and measure how much the model's score drops. The drop is the feature's importance. Slower than tree importance but more honest (it does not get fooled by correlated features the way Gini-based importance does).

In practice, the production pipeline is usually: filter (variance threshold to drop trivially useless features) → embedded (Lasso or tree importance to drop weak features) → keep what matters.

Watch L1 regularization perform automatic feature selectionInteractive

Loading visualization...

#Data Leakage: The Cardinal Sin

Every senior ML interviewer has the same favorite trap: ask the candidate to describe their preprocessing pipeline and listen for data leakage. Most candidates fail.

Data leakage is when information from outside the training set sneaks into training, inflating evaluation scores in a way that does not generalize to production. The model looks great offline and craters online.

#The Most Common Leak: Fitting on Full Data

pythonrunnable cell

# BROKEN
scaler = StandardScaler().fit(X)            # uses test statistics!
X_scaled = scaler.transform(X)
X_train, X_test = train_test_split(X_scaled, ...)

The scaler's mean and standard deviation were computed using test data. Information about the test distribution leaked into the transformation. Your cross-validation score is optimistic.

The fix:

pythonrunnable cell

# CORRECT
X_train, X_test = train_test_split(X, ...)
scaler = StandardScaler().fit(X_train)      # train only
X_train_scaled = scaler.transform(X_train)
X_test_scaled  = scaler.transform(X_test)   # apply train stats to test

This applies to every learned transformation: imputers, encoders, scalers, target encoders, PCA, anything with a fit method. Fit on train only.

#Target Leakage

A more insidious leak: a feature that looks innocent but is computed using the target.

"Mean amount of past transactions for this user" — leaks if you include the current transaction's amount in the average.
"Did this customer churn next month?" — accidentally included as a feature when it is supposed to be the target.
Target encoding without cross-fold — each row's encoded value uses the row's own target.

These are nasty because the model train/test split does not catch them. The test set has the same leak as the train set. Only production reveals the bug.

#Time Leakage

For any problem with time, you must split temporally — train on the past, test on the future. Random shuffling lets information from the future bleed into training:

Computing a 30-day rolling average using future days as well as past days.
Including a feature like "user's total lifetime spend" that includes spend from after the prediction date.
Cross-validating with KFold instead of TimeSeriesSplit on time-series data.

What Do You Think?

Which of these is data leakage?

#Pipeline Discipline

The single most important habit to develop: wrap every transformation in a sklearn Pipeline, and use ColumnTransformer to apply different transformations to different columns. There is no reason to do this manually. Pipelines:

Prevent leakage — pipeline.fit(X_train) fits every step on training data; pipeline.transform(X_test) reuses the fit. Impossible to accidentally fit a scaler on test data.
Cross-validate correctly — cross_val_score(pipeline, X, y, cv=5) fits the entire pipeline (scalers, encoders, model) inside each fold. Manual preprocessing followed by cross_val_score(model, ...) leaks across folds.
Serialize cleanly — one joblib.dump(pipeline, "model.pkl") saves the whole transform-and-predict path. Production inference is one pipeline.predict(new_row) call.

Let's build one.

Loading visualization...

#Showing the Leak in Action

Theory is one thing. Let's actually watch what happens when target encoding is done wrong vs. right. The cell below computes target encoding two ways: (1) using the full training set (the classic bug), (2) using out-of-fold computation. The cross-validation score for the broken version will be optimistically inflated.

Loading visualization...

The gap between the two AUC bars is the "lie" — the amount your model would appear to have improved if you had not noticed the leak. In a real Kaggle context, this lie has cost real teams real prize money when their public leaderboard score (computed on leaked features) collapsed on the private leaderboard.

Quick check

You are one-hot encoding a 'color in {red, green, blue}' feature for an unregularized linear regression. Why drop one column?

#Putting It All Together: A Pragmatic Checklist

When you sit down with a new tabular dataset, the order of operations:

Explore. df.info(), df.describe(), df.isna().sum(), histograms. Get a feel for distributions and missingness before you touch anything.
Split first. train_test_split before any transformation. Never let a fitted transformer see test data.
Decide per column.
- Numerical, roughly Gaussian → StandardScaler.
- Numerical, skewed → log/sqrt or Yeo-Johnson, then StandardScaler.
- Numerical, outlier-heavy → RobustScaler.
- Numerical, you want non-linearity in a linear model → bin it.
- Categorical, cardinality < 50 → OneHotEncoder.
- Categorical, cardinality 50–10,000 → TargetEncoder (with cv).
- Categorical, cardinality > 10⁶ → FeatureHasher.
- Date → expand to components + cyclical encoding.
- Text → TF-IDF + char n-grams (or BERT embeddings, see NLP track).
- Missing → is_missing flag + imputer; or leave NaN alone for XGBoost/LightGBM.
Wrap in a Pipeline. Use ColumnTransformer to apply different sub-pipelines to different columns. The full chain must end in the model.
Cross-validate the pipeline, not the model. — never fit transforms outside CV.

#Key Takeaways

Features matter more than models for tabular ML. Every Kaggle winner, every production fraud team, every churn analyst spends 80% of their time on features. The model is a commodity.
Scale what needs scaling. Linear models, SVMs, k-NN, k-means, PCA, neural networks all need scaled inputs. Tree models do not. StandardScaler is the default; RobustScaler for outliers; MinMaxScaler for bounded ranges; Normalizer for direction-only vectors.
Right-skewed numerical features benefit from log, sqrt, Box-Cox, or Yeo-Johnson. Yeo-Johnson handles zero and negative numbers natively — reach for it when Box-Cox can't.
Categorical encoding depends on cardinality. One-hot for low, target encoding (with smoothing + out-of-fold) for medium-high, hashing trick for billions. Ordinal only when there is real order.
Time features need to be expanded into hour/day/month/weekend/holiday, plus cyclical sin/cos for periodic structure.
Missingness is often a feature itself. Add is_missing flags before imputing. XGBoost/LightGBM/CatBoost handle NaN natively — for them, don't impute at all.
Leakage is the cardinal sin. Always fit transformers on training data only. Target encoding without cross-fold is the most common interview trap and the most common production-killing bug. Wrap everything in a sklearn Pipeline and the leak becomes architecturally impossible.

Anecdote

Historical Note

The history of feature engineering tracks the history of applied statistics itself. Box and Cox introduced the Box-Cox transform in 1964 specifically to make skewed data look Gaussian — they were trying to make linear regression's normality assumption hold. Yeo and Johnson extended it to negative numbers in 2000. The "hashing trick" was popularized by Weinberger, Dasgupta, Langford, Smola, and Attenberg in their 2009 paper "Feature Hashing for Large Scale Multitask Learning," which made click-through-rate prediction at web scale possible. Target encoding (then called "impact coding") was formalized by Daniele Micci-Barreca in 2001 specifically for high-cardinality categoricals in credit scoring, and it became the secret weapon of Kaggle's top performers a decade later. The thread connecting all of these: someone had real data, a model that wouldn't fit it, and a few months of clever statistical thinking that turned a useless feature into a useful one.

You now know how to scale, encode, transform, impute, and select features without leaking your test set into training. The next step is making the model itself robust enough that your feature engineering pays off: Cross-Validation & Model Selection — the discipline of evaluating a pipeline honestly when there is no production traffic yet to tell you whether you have lied to yourself.

cross_val_score(pipeline, X, y, cv=...)

Your Reflection

Continue Learning

Discussion

Feature Engineering: The 80% of ML

#Why Features Matter More Than Models

#Numerical Features: Scaling

#StandardScaler (z-score)

#MinMaxScaler

#RobustScaler

#Normalizer

#Numerical Features: Skew Transformations

#Log Transform

#Square Root

#Box-Cox

#Yeo-Johnson

#Numerical Features: Binning

#Numerical Features: Interactions

#Categorical Features

#One-Hot Encoding

#Ordinal Encoding

#Target / Mean Encoding (with smoothing and cross-fold)

#Frequency Encoding

#Hashing Trick

#Embeddings (Forward Reference)

Deep Dive

#Text Features (Quick Coverage)

#Date/Time Features

#Missing Data

#Imputation Strategies

#The "Missingness as a Feature" Trick

#When to Leave NaN Alone

#Feature Selection

#Filter Methods

#Wrapper Methods

#Embedded Methods

#Data Leakage: The Cardinal Sin

#The Most Common Leak: Fitting on Full Data

#Target Leakage

#Time Leakage

#Pipeline Discipline

#Showing the Leak in Action

#Putting It All Together: A Pragmatic Checklist

#Key Takeaways

Deep Dive