"In 2003, NeurIPS ran a feature-selection challenge where the winning team beat the baseline by 30% — not by changing models, but by throwing away 95% of the features. The blessing of dimensionality is a myth: extra features add noise, multicollinearity, and training cost. A great ML engineer asks 'which features should I remove?' before 'which model should I try?'"
Learning Objectives
After this lesson, you will be able to:
Explain why fewer features often mean better generalization, faster training, and easier explainability — and why dumping every available column into a model is a beginner mistake
Apply filter methods (variance threshold, correlation filter, mutual information, chi-squared, ANOVA F-test) to score features against a target before any model training
Use wrapper methods (Recursive Feature Elimination, sequential forward/backward selection) when accuracy matters more than compute and you can afford repeated model fits
Use embedded methods (L1/Lasso, tree feature importance, Boruta) that select features as a side effect of fitting — and know why permutation importance beats Gini importance for any tree-based model
Build this --> Take any messy dataset with 50+ columns -- a Kaggle dataset, your own logs, a public CSV -- and write a script that ranks every feature by mutual information against the target, then trains two models: one on all features and one on the top 10; compare accuracy, training time, and inference latency
Don't worry if "filter vs wrapper vs embedded" sounds bureaucratic — once you see them in code, the trade-off (speed vs accuracy vs how-much-do-you-trust-your-model) becomes obvious.
The curse of dimensionality is the real reason this matters. As feature count p grows, the volume of feature space grows exponentially. With finite data n, your samples become sparse in that space, distance metrics break down, and any model with enough capacity will find spurious patterns. The classic warning sign: training accuracy keeps climbing as you add features while validation accuracy plateaus or drops. That gap is overfitting, and it grows with every useless feature you keep.
Filter methods rank each feature by a statistical score against the target, completely independent of which model you plan to train. They are fast (linear in n × p), parallelizable, and a great first pass on any dataset with hundreds or thousands of columns.
The cheapest filter of all: drop any feature whose values barely change. A feature with variance near zero is a constant in disguise -- it cannot help any model split groups apart.
Variance threshold catches things like a "country" column that is 99.8% USA, or a sensor reading stuck at zero. It catches nothing about predictive value -- a constant feature can still correlate with the target (it cannot, actually, but you get the point). Use it as a sanity sweep, not a primary tool.
For numeric features, drop one of any pair whose Pearson correlation |r| exceeds 0.95 (or whatever threshold makes sense). They carry the same information; keeping both adds compute without adding signal and destabilizes linear-model coefficients.
Mutual information measures how much knowing one variable reduces uncertainty about another. Unlike Pearson correlation, it captures non-linear relationships -- a U-shaped or step-function dependence registers as high MI even when r ≈ 0.
I(X;Y)=x,y∑p(x,y)logp(x)p(y)p(x,y)
Mutual information is the single highest-leverage filter for tabular ML. It is non-parametric, it handles non-linearity, and sklearn's implementation works for both classification (mutual_info_classif) and regression (mutual_info_regression). Spend the 60 seconds it takes to run on every new dataset.
For categorical features vs. categorical target, use chi-squared (chi2) -- it tests whether the joint distribution of feature and target differs from independence.
For numeric features vs. categorical target, use ANOVA F-test (f_classif) -- it tests whether the feature's mean differs across target classes.
These are faster than mutual information but assume linear-ish relationships. Use them when you have thousands of features and need a rapid pre-filter; use mutual information when you can spend the extra seconds.
What Do You Think?
You have a tabular classification dataset with 10,000 features and only 500 samples. What should be your FIRST step before any modeling?
With 10,000 features and 500 samples (p >> n), you are deep in the curse-of-dimensionality regime. Wrapper methods (RFE-CV) require fitting your model hundreds of times -- each fit overfits because of the sample shortage, so the rankings are noise. Embedded methods like XGBoost still need to look at all 10,000 features; some will look "important" by random chance. PCA destroys interpretability and may not even reduce overfitting. The right first move is a fast filter: rank by mutual information, keep the top 200-500, then apply wrapper or embedded selection on the survivors.
Wrapper methods evaluate feature subsets by actually training the model on each subset and measuring cross-validated performance. They are model-aware (RFE on a Random Forest finds different features than RFE on logistic regression) and slow (you fit your model O(p) times or worse).
Rank features by importance (model coefficient magnitude, tree feature importance).
Drop the bottom k features.
Repeat until you reach the target feature count.
Try it: Score features by importance and watch the impact of dropping eachInteractive
Loading visualization...
RFE with cross-validation (RFECV) automatically picks the optimal feature count by tracking validation score across removal steps. The price: you fit your model p / k times for each CV fold. On a Random Forest with 1000 features and 5-fold CV, that is hundreds of fits.
Forward: Start with zero features. At each step, add the feature that most improves CV score. Stop when no addition helps.
Backward: Start with all features. At each step, drop the feature whose removal most improves (or least hurts) CV score. Stop when removal hurts.
These greedy algorithms can miss feature combinations that only help together (e.g., feature_a × feature_b is informative but neither alone is). They are also expensive. Use them when accuracy is paramount and compute is cheap -- production deployment, final Kaggle submission, regulated medical work.
Embedded methods bake feature selection into model fitting itself. No separate scoring phase, no model-fits-per-feature explosion -- just train the model and read off which features it kept.
Add an L1 penalty to the loss function. The optimizer is incentivized to drive coefficients to exactly zero (not just small) -- selecting features by side effect.
Llasso(β)=ordinary least squares2n1i=1∑n(yi−β0−xi⊤β)2+L1 penaltyαj=1∑p∣βj∣
Two important details:
Scale your features first. L1 penalizes coefficient magnitudes, so unscaled features (one in dollars, one in cents) get unfair penalties. Always run StandardScaler before Lasso.
Use LassoCV to pick α. It runs cross-validation across a path of α values and picks the one minimizing CV error. Setting α by hand is guesswork.
ElasticNet blends L1 and L2 penalties and works better when features are correlated -- L1 alone tends to arbitrarily pick one feature from a correlated group, while ElasticNet shares weight more stably.
Random Forests, Gradient Boosting (XGBoost, LightGBM, CatBoost), and Extra Trees all expose a feature_importances_ attribute after fitting. These are usually Gini importance (for classification) or mean decrease in impurity -- the average improvement to split criterion that each feature provides, weighted by how often it is used.
It is convenient. It is also biased toward high-cardinality features (continuous variables and categoricals with many levels), which is a serious problem.
Use permutation importance as your primary tool for any tree-based model. It works on the validation set (so it reflects generalization importance, not memorization), it is unbiased across feature types, and it does not require refitting -- only re-scoring with one column shuffled at a time.
Boruta wraps a Random Forest and compares each real feature's importance against a "shadow" feature (the same column with values shuffled). A feature passes if it consistently beats its shadow. Aggressive but principled.
SHAP-based selection uses SHAP values (game-theoretic feature attributions) to rank features by total contribution magnitude. Slower than Gini, more honest, and explains why the model uses each feature.
Most production ML teams converge on a layered approach:
Cheap filter sweep. Variance threshold (drop near-constants) → correlation filter (drop near-duplicates) → mutual information ranking (keep top 30-50% by MI score). Takes seconds.
Embedded selection during training. Fit a regularized linear model (Lasso) or a gradient-boosted tree on the survivors. Read off coefficients or feature importances.
Permutation importance audit on the validation set. Confirm that "important" features actually generalize. Drop any feature whose permutation importance is statistically indistinguishable from zero.
Wrapper only if accuracy critical. RFE-CV on the survivors of step 3 -- maybe 30 features down to the optimal 12.
This sequence cuts compute aggressively early (filter) and reserves the expensive analysis for the small surviving feature set (wrapper).
Try it: Compare scaler choices and watch how feature distributions and outliers behave under eachInteractive
Loading visualization...
(Reminder: scaling matters for L1-based selection. Without it, large-magnitude features dominate the penalty term and small-magnitude features get unfairly zeroed out regardless of predictive value.)
Tests · Verify that mutual information, L1 coefficients, and permutation importance agree on the most predictive features. Verify that the top-5 model performs at least as well as the all-features model.
Selection inside the CV fold. If you select features (filter, wrapper, or embedded) using the full dataset and then cross-validate, your CV scores are optimistic by exactly the amount the selection used the held-out folds. Always wrap selection in a Pipeline that lives inside cross_val_score.
Correlated features confuse importance. If feature_a and feature_b are 95% correlated, one model fit may give all the weight to a and another to b, even though both are equally useful. Use permutation_importance with n_repeats >= 10 to average across runs, or group correlated features and treat them as a unit.
Train/serve skew of the feature set. The feature list is a contract. If your training pipeline selects different features than the serving pipeline expects (e.g., because selection ran on different data), predictions silently break. The selected feature list must be versioned and checked at inference time.
Fewer features almost always beat more features. Every irrelevant column adds compute, hurts generalization (curse of dimensionality), complicates monitoring, and gives the model another chance to overfit to noise; selection is a real first-class step, not a polish task
Filter, wrapper, embedded are speed/accuracy/honesty trade-offs. Filter (mutual information, ANOVA) is fast and model-agnostic; wrapper (RFE) is accurate but expensive; embedded (Lasso, tree importance) is the practical default because it piggy-backs on training you would do anyway
Mutual information is the highest-leverage filter. It captures non-linear relationships that Pearson correlation misses, runs in seconds, and works for both classification and regression; run it on every new dataset before doing anything else
Permutation importance beats Gini importance for trees. Gini is biased toward high-cardinality features and can rank noise features above genuine signal; permutation importance is unbiased, model-agnostic, computed on a held-out set, and only requires re-scoring (not re-fitting)
Selection must live inside the CV loop. Scoring features against the target on the full dataset and then cross-validating leaks the test set into the model; wrap selection in a sklearn Pipeline so it runs fresh inside each fold
You have a dataset with 5,000 features and 800 samples. What is the best opening move?
You now know which features are worth keeping. Next up: Dimensionality Reduction — when even the surviving feature set is too big or too correlated, how to project it into a smaller space (PCA, t-SNE, UMAP) without losing the signal.