"In Kaggle's 2024 ML practitioner survey, 'feature engineering' beat 'model selection' as the top factor in winning models. Most production ML systems still get 80% of their gains from features, not architecture. This is where ML engineering separates from ML academia — and where a thoughtful data scientist with XGBoost still routinely beats a careless one with a transformer."
Learning Objectives
After this lesson, you will be able to:
Rescale numbers using min-max normalization, standard scaling, and robust scaling so big and small features play fair in distance-based and gradient-based models
Encode categorical variables with one-hot, ordinal, and target encoding — and know which to use based on cardinality and the risk of data leakage
Create interaction features by multiplying, dividing, and combining existing columns to surface relationships that linear models cannot discover on their own
Build polynomial features for nonlinear relationships, apply cyclical sin/cos encoding for time-based features, and select the most predictive features to reduce overfitting
Don't worry if "feature engineering" sounds intimidating — it is really just creating new columns from your existing data. Once you see a few examples, you will wonder why it has such a fancy name!
Feature engineering is widely considered the most important skill in applied machine learning. Andrew Ng has said: "Applied machine learning is basically feature engineering." Kaggle grandmasters consistently report that feature engineering accounts for more of their winning performance than model selection or hyperparameter tuning.
Tests · Verify the DataFrame has more columns after engineering. Verify revenue equals price times quantity. Verify is_weekend is binary.
Try it! Open the Python REPL and type these lines yourself. Create a quick DataFrame and engineer one feature: import pandas as pd; df = pd.DataFrame({'date': pd.date_range('2024-01-01', periods=7)}); df['day_of_week'] = df['date'].dt.day_name(); print(df) — you just did feature engineering!
Models that compute distances (KNN, SVM) or use gradient-based optimization (linear regression, neural networks) are sensitive to feature scales. A feature ranging from 0 to 1,000,000 will dominate a feature ranging from 0 to 1. Scaling fixes this.
Uses median and IQR instead of mean and standard deviation.
xscaled=Q3−Q1x−median
Use when: Your data has significant outliers that you want to keep but not let them distort the scaling.
What Do You Think?
You have a feature 'annual_income' with values ranging from $15,000 to $50,000,000 (a few billionaires in the dataset). Which scaling method should you use?
With extreme outliers, min-max would compress 99.9% of the data into a tiny range near 0, and standard scaling would be distorted by the extreme mean and standard deviation. Robust scaling uses the median and IQR, which are barely affected by billionaire outliers.
Scaling Comparison — Toggle Outliers and Watch Each Scaler ReactInteractive
Models need numbers, not strings. Encoding converts categorical variables into numeric representations.
This section is a quick recap. For the full treatment — target encoding with K-fold to prevent leakage, frequency/hashing for high-cardinality, and learned embeddings — see the dedicated Categorical Encoding lesson, which is now a prerequisite to this one.
Use when: Unordered categories with low cardinality (<50 unique values). Beware: High cardinality creates sparse, high-dimensional data (a city column with 10,000 cities creates 10,000 new columns).
Replaces each category with the mean of the target variable for that category.
xencoded=nc1i∈c∑yi
Use when: High cardinality categories where one-hot encoding is impractical. Danger: High risk of data leakage and overfitting. Must use regularization (smoothing, leave-one-out) and compute only on training folds.
#Feature Interactions: Where the Real Signal Hides
Sometimes the signal is not in individual features but in their COMBINATION. Price alone does not predict sales. Quantity alone does not predict sales. But price x quantity = revenue, and THAT predicts profit. Always ask: do any features multiply, divide, or combine to create a stronger signal?
Common interaction patterns:
Multiplication: revenue = price x quantity, area = length x width
Conditional: high_value_weekend = (is_weekend) x (order_value > 100)
The key insight: linear models cannot discover interactions on their own. If you do not create the interaction feature, a linear model literally cannot learn that relationship. Tree-based models can discover some interactions through nested splits, but explicit interaction features still help them learn faster and with less depth.
You start with the raw columns from your dataset -- dates as strings, categorical columns as text, numerical values at wildly different scales, and missing values scattered throughout. These raw features are what your data pipeline produced, but they are not yet suitable for most models.
Before any transformation, decide how to handle nulls. Impute numerical columns with the median (robust to outliers), categorical columns with the mode, or create "is_missing" indicator columns for MNAR data. Imputation statistics must come from the training set only.
Apply min-max normalization for bounded outputs, standard scaling (z-score) for roughly normal data, or robust scaling (median/IQR) when outliers are present. Tree-based models skip this step, but linear models and neural networks require it.
Convert strings to numbers. Use one-hot encoding for low-cardinality unordered categories, ordinal encoding for ordered categories (small/medium/large), or target encoding with regularization for high-cardinality features like zip codes.
Engineer new features that capture domain knowledge: price_per_sqft from price and area, BMI from height and weight, days_since_last_purchase from timestamps. Polynomial features and cyclical sin/cos encoding for time features go here too.
Not all features help. Remove zero-variance features, drop one of any highly correlated pair (r > 0.95), and use tree-based importance or mutual information to rank the rest. Validate with cross-validation that removing features does not degrade performance.
The output is a clean, scaled, encoded, and selected feature matrix -- ready for model training. Every transformation is encapsulated in a reproducible pipeline that applies identically during training and inference, preventing train-serve skew.
Not all features help. Some are redundant, some are noisy, and some are correlated with the target only by coincidence.
Filter methods: Rank features by statistical tests (correlation, chi-squared, mutual information) and keep the top k.
Wrapper methods: Train models with different feature subsets, keep the best. Computationally expensive but thorough.
Embedded methods: Let the model itself select features during training. L1 regularization (Lasso) drives unimportant feature weights to exactly zero. Tree-based models provide feature importance scores.
Tests · Verify pricePerSqft is computed correctly. Verify standard scaling produces mean ~0. Verify one-hot encoding sums to 1 per row. Verify cyclical month encoding.
Feature engineering is the most impactful skill in applied ML. Domain knowledge translated into features accounts for more performance gain than model selection or hyperparameter tuning, especially for tabular data
Different model families need different preprocessing. Tree-based models need no scaling and handle categoricals natively, while linear models and neural networks require scaling and encoding; always match your preprocessing to your model
Cyclical features need sin/cos encoding. Integer encoding treats December (12) and January (1) as distant when they are adjacent; sin/cos encoding preserves the circular structure that time-based patterns require
Domain-specific derived features unlock hidden signal. Raw timestamps, prices, and measurements are rarely useful directly; features like price-per-square-foot, days-since-last-purchase, and rolling averages capture the relationships models need
Feature selection reduces overfitting. Too many features cause the model to memorize noise; remove zero-variance, highly correlated, and low-importance features, then validate with cross-validation
Why do tree-based models (Random Forest, XGBoost) NOT need feature scaling?
Modern feature engineering for ML in 2026 is increasingly automated -- AutoML and feature stores (Feast, Tecton) handle scaling, encoding, and lag generation -- but the decisions still belong to you: which transforms preserve signal, which leak, and which scale a model from "good" to "deployable."
Feature engineering transforms data from raw to powerful. But what if you do not have enough data? Next up: Data Augmentation -- techniques to generate more training data from what you already have.