RugvAI Labs

Regularization: L1, L2, and ElasticNet

Your perfect-fit model is broken. Coefficients of +7213, -41827, +89104 sum to a great training accuracy and predict nothing useful. Regularization fixes this in 2 lines of code. Here's the intuition that makes you reach for it without thinking.

Learning Objectives

After this lesson, you will be able to:

Understand L2 regularization as putting a 'budget cap' on your model's weights — it cannot spend too much on any one feature
Understand why L1 regularization sets useless features to exactly zero — automatic feature selection built into the math
Pick the right regularization method (L1, L2, or both via ElasticNet) for your specific problem
Tune the regularization strength (lambda) to find the sweet spot between underfitting and overfitting

Prerequisites

Before You Start

Review

Why

Why This Matters

Every production ML model uses regularization -- without it, your model memorizes the training data perfectly but fails on new data; regularization is the difference between a model that works on your laptop and one that works in the real world
L1 is how automatic feature selection happens -- when you have 1000 features and most are noise, L1 regularization zeros out irrelevant features automatically; it is the built-in feature selector that every data scientist should reach for before running manual selection
Weight decay is the neural-network cousin of L2 -- the weight_decay parameter in AdamW (used by PyTorch, every transformer model, every LLM fine-tuning run) shrinks weights toward zero on the same intuition you will build here. They are not the same operation, though: AdamW exists precisely because adding an L2 term to the loss behaves differently under an adaptive optimizer than decaying the weights directly (see the deep-learning track for the decoupled update). Understanding this lesson means you understand the most critical training knob in deep learning
Build this --> Build a polynomial regression pipeline: fit a degree-10 polynomial with no regularization (watch it overfit), add L2 (watch it smooth out), then add L1 (watch coefficients go to zero one by one as you increase λ); visualize the regularization path

Don't worry if regularization feels abstract — the core idea is simple: it stops your model from going overboard. Think of it as a teacher saying "keep your answer simple" instead of letting you write a 50-page essay for a 5-point question!

Active Recall

Before we go further, recall: in the bias-variance tradeoff lesson, what was the SIGNATURE of a high-variance (overfit) model — what specifically goes wrong on train vs test, and why? Write one sentence before reading on. (Forcing yourself to retrieve this now is more durable than re-reading it.)

Write your answer in your own words — don't look back at the lesson. This is the most effective way to remember what you just learned.

Type your explanation above

#War Story: The Perfect-Fit Model That Was Actually Broken

A data science intern at an e-commerce company shipped a polynomial regression to predict customer lifetime value. Training R² was 0.998. The model fit every dot in the cloud almost exactly. Two weeks later, the predictions in production looked random — RMSE on new customers was 3.4x worse than a simple linear baseline.

The coefficients of the "perfect" model:

w_1 =  +7,213.4
w_2 = -41,827.9
w_3 = +89,104.2
w_4 = -78,910.6
w_5 = +34,521.3

Five weights, four with alternating signs, magnitudes in the tens of thousands. Each weight wasn't predicting anything real — they were collectively cancelling out to fit the training noise. Any new data point that landed even slightly between training rows produced an absurd extrapolation (predictions of -$200,000 LTV on real customers).

The fix was literally two lines:

pythonrunnable cell

from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)  # replaces LinearRegression

After scaling features (critical — see the CommonMistake below), Ridge produced weights all under 100 in magnitude, training R² dropped to a "merely OK" 0.84, and production RMSE matched the offline test set. The model that fit worse in training worked dramatically better in production. That's the bias-variance tradeoff from the previous lesson, weaponized into a single hyperparameter alpha.

This lesson is about turning that knob deliberately.

#The Overfitting Problem

Overfitting happens when a model learns not just the true signal in the data, but also the random noise specific to the training set. The symptoms are clear:

Training error: very low (model memorizes every training point)
Test error: much higher (the memorized noise does not generalize)
Model coefficients: large in magnitude, often with alternating signs that cancel each other out

A model with weights w = [4200, -4198, 3.2] has almost certainly overfit. The first two weights nearly cancel — they are fitting noise. Regularization prevents this by penalizing large weight magnitudes directly in the loss function.

Quick check

You train a regression model and observe: training MSE = 0.02, test MSE = 4.50. Which is the BEST diagnosis?

#L2 Regularization (Ridge)

#The Loss Function

Ridge regression adds the sum of squared weights to the loss:

L_{\text{Ridge}}(\mathbf{w}) = \underbrace{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2}_{\text{MSE}} + \underbrace{\lambda \sum_{j=1}^{d} w_j^2}_{\text{L2 penalty}}

Try it! Open the Python REPL and type these lines yourself. Compare Ridge (L2) vs. plain regression: from sklearn.linear_model import LinearRegression, Ridge; import numpy as np; X = np.random.randn(20,5); y = np.random.randn(20); print("No regularization:", LinearRegression().fit(X,y).coef_.round(2)); print("With L2 (Ridge): ", Ridge(alpha=10).fit(X,y).coef_.round(2)) — notice how Ridge shrinks all the weights!

#The Gradient

\nabla_{\mathbf{w}} L_{\text{Ridge}} = -\frac{2}{n} \mathbf{X}^\top(\mathbf{y} - \hat{\mathbf{y}}) + 2\lambda \mathbf{w}

The update rule reveals why L2 is also called weight decay: at each step, weights are multiplied by (1 - 2·α·λ) — a factor slightly less than 1. Every weight shrinks proportionally to its magnitude. This never forces weights to exactly zero; it just makes them smaller.

#Geometric Intuition: The Sphere Constraint

The L2 penalty is equivalent to adding a spherical (circular in 2D) constraint on the weight vector. The optimization problem becomes:

Minimize MSE subject to ||w||² ≤ t (budget constraint)

#The Unconstrained Solution

Without regularization, the optimal weights sit at the center of the MSE loss contours (the ellipses). Call this point w*. The model can set weights to any values — and it will often choose large ones to chase noise in the training data.

#Adding the L2 Constraint (Sphere)

The L2 constraint ||w||² ≤ t defines a sphere (circle in 2D) centered at the origin. The solution must live inside this sphere. As λ increases, t decreases — the sphere shrinks.

#Where the Ellipses Touch the Sphere

The regularized solution is where the MSE loss contours first touch the sphere. Because a sphere has no corners, this contact point almost never lands exactly on an axis. Therefore, L2 regularization shrinks weights toward zero but rarely makes them exactly zero.

#The Result: Smooth Shrinkage

Key property of L2: All features are kept in the model. Weights approach zero but never reach it (unless λ → ∞). This makes L2 the right choice when you believe most features are genuinely relevant.

pythonplayground.py · Pyodide

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# Generate polynomial regression problem (easy to overfit)
np.random.seed(42)
X = np.sort(np.random.uniform(-3, 3, 100)).reshape(-1, 1)
y = np.sin(X).ravel() + np.random.normal(0, 0.3, 100)

# Polynomial features (degree 10 -- prone to overfit)
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=10, include_bias=False)
X_poly = poly.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(X_poly, y, test_size=0.2, random_state=42)

# Scale BEFORE regularization (critical -- see CommonMistake below)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Try different regularization strengths
lambdas = [0.0001, 0.01, 1.0, 10.0, 100.0]
for lam in lambdas:
  model = Ridge(alpha=lam)
  model.fit(X_train_scaled, y_train)
  train_mse = np.mean((model.predict(X_train_scaled) - y_train) ** 2)
  test_mse = np.mean((model.predict(X_test_scaled) - y_test) ** 2)
  max_coef = np.max(np.abs(model.coef_))
  print(f"lambda={lam:8.4f} | train_mse={train_mse:.4f} | test_mse={test_mse:.4f} | max_coef={max_coef:.2f}")

#L1 Regularization (Lasso)

#The Loss Function

Lasso (Least Absolute Shrinkage and Selection Operator) uses the sum of absolute values instead of squared values:

L_{\text{Lasso}}(\mathbf{w}) = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{d} |w_j|

#The Gradient (Subgradient)

\frac{\partial L_{\text{Lasso}}}{\partial w_j} = -\frac{2}{n}\sum_{i=1}^n (y_i - \hat{y}_i)x_{ij} + \lambda \cdot \text{sign}(w_j)

The key difference from L2: the penalty is λ·sign(w) — a constant push toward zero regardless of the weight's magnitude. For L2, the push is 2λ·w — proportional to the weight. With L2, as a weight approaches zero, the push weakens and never crosses zero. With L1, the push is always the same constant λ — small weights get pushed just as hard as large ones and cross zero, becoming exactly zero.

#Geometric Intuition: The Diamond Constraint

The L1 constraint ||w||₁ ≤ t (sum of absolute values) defines a diamond shape (L1 ball) in weight space:

#The Unconstrained Solution

The unregularized solution w* sits at the center of the MSE loss ellipses. It may have many non-zero components — all the noise-fitting coefficients that overfit the training data.

#The L1 Constraint is a Diamond

The L1 ball {w : |w₁| + |w₂| ≤ t} forms a diamond (rotated square in 2D, a cross-polytope in higher dimensions). Its key geometric feature: pointed corners that align exactly with the coordinate axes — points where one weight is non-zero and all others are zero.

#The Loss Contours Hit a Corner

As λ increases, the diamond shrinks. The first point where the MSE ellipses touch the diamond is almost always at a corner — a point on a coordinate axis. At a corner, exactly one (or a few) weights are non-zero; all others are exactly zero.

#Sparsity: Some Features Become Exactly Zero

Key property of L1: Some weights become exactly zero. The model performs automatic feature selection — irrelevant features are zeroed out completely. This is ideal for high-dimensional problems where most features are noise.

Quick check

Why does L1's diamond-shaped constraint region produce SPARSE solutions while L2's spherical region does not?

#Concrete example: L1 on 5 features

Suppose we fit Lasso to predict customer churn on 5 features. Here's what the coefficient vector looks like at three different λ values:

Feature	λ = 0 (no reg)	λ = 0.1	λ = 1.0
`months_active`	+0.84	+0.71	+0.42
`last_login_days`	-0.61	-0.48	-0.19
`support_tickets`	+0.33	+0.12	0.00
`random_uuid_hash`	-0.21	0.00	0.00
`mouse_x_avg_px`

Read from right to left: at strong regularization (λ=1.0), only the two genuinely predictive features survive. The noise features (random_uuid_hash, mouse_x_avg_px) are eliminated at moderate regularization (λ=0.1). The marginal feature (support_tickets) survives moderate regularization but is eliminated under strong regularization. This is what we mean by "automatic feature selection."

Interactive Lab

Drag the lambda slider and see exactly which features survive Lasso and which die. Compare side-by-side with Ridge — same lambda, totally different behavior at the extremes.

Watch coefficients shrink to zero as λ grows

The animation below shows the same idea inline: trace each coefficient as λ grows. Watch how the Lasso path snaps individual weights to exactly zero, while the Ridge path slides them smoothly toward zero without ever crossing.

Ridge vs Lasso regularization pathInteractive

Loading visualization...

What Do You Think?

You have 1000 features predicting house price. You suspect only about 50 actually matter. Which regularization should you use?

L1/Lasso is the right answer here. When you have 1000 features and expect only ~50 to matter, you want a method that zeros out the irrelevant 950. L1 does this automatically. L2 keeps all 1000 features with small weights — the model is harder to interpret and uses more memory at inference time. L1 effectively performs feature selection as part of training, giving you a sparse model that is both interpretable and efficient.

What Do You Think?

Your Ridge regression model has very low training error but much higher test error. You decide to increase λ. What happens to training error?

Training error increases when you increase λ. This is the bias-variance tradeoff in action. Increasing λ adds more bias (the model is constrained to use smaller weights), which means it cannot fit the training data as precisely. The goal is to increase λ just enough that the reduction in variance (better generalization) outweighs the increase in bias (worse training fit). The optimal λ is found by cross-validation.

#ElasticNet: The Best of Both

When features are correlated, Lasso has a problem: it arbitrarily picks one feature from a correlated group and zeros out the others. This makes the solution unstable — small changes in data can cause it to pick a different feature. ElasticNet fixes this by combining L1 and L2:

L_{\text{ElasticNet}}(\mathbf{w}) = \text{MSE} + \lambda_1 \sum_{j=1}^d |w_j| + \lambda_2 \sum_{j=1}^d w_j^2

ElasticNet has two hyperparameters:

λ₁ (L1 ratio in sklearn): Controls sparsity. Higher → more zeros.
λ₂ (alpha·(1 - l1_ratio) in sklearn): Controls weight shrinkage. Higher → smaller weights.

When to use ElasticNet

Features are correlated and you want to select groups, not individuals
You are unsure whether L1 or L2 is better (ElasticNet interpolates)
The number of features is larger than the number of samples (p >> n)
Genomics, text, and financial data with many correlated signals

Time to see all three penalties side by side. The cell below builds a 100-feature regression problem where only 10 features are truly informative, then fits Ridge, Lasso, and ElasticNet -- so you can count exactly how many coefficients each one zeros out.

Loading visualization...

What Do You Think?

In the playground above (sparse problem with 10 of 100 features informative), which model do you expect to zero out the MOST coefficients?

#Choosing λ: The Regularization Path

The strength of regularization λ is a hyperparameter — it must be tuned, not learned. The standard method is cross-validation: try many values of λ, evaluate each on a validation set, pick the λ that minimizes validation error.

pythonplayground.py · Pyodide

import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge, Lasso, RidgeCV, LassoCV
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.datasets import make_regression

# Create a sparse regression problem
# True model: only 10 of 100 features actually matter
X, y, true_coef = make_regression(
  n_samples=200,
  n_features=100,
  n_informative=10,
  noise=20,
  coef=True,
  random_state=42
)

# Scale features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Regularization path: how do coefficients change as lambda increases?
lambdas = np.logspace(-2, 4, 100)

ridge_coefs = []
lasso_coefs = []

for lam in lambdas:
  ridge = Ridge(alpha=lam).fit(X_scaled, y)
  lasso = Lasso(alpha=lam, max_iter=5000).fit(X_scaled, y)
  ridge_coefs.append(ridge.coef_)
  lasso_coefs.append(lasso.coef_)

ridge_coefs = np.array(ridge_coefs)
lasso_coefs = np.array(lasso_coefs)

# Count non-zero coefficients for each lambda
ridge_nonzero = np.sum(np.abs(ridge_coefs) > 0.01, axis=1)
lasso_nonzero = np.sum(np.abs(lasso_coefs) > 0.01, axis=1)

print("Lambda      | Ridge non-zero | Lasso non-zero")
print("-" * 45)
for i in [0, 20, 40, 60, 80, 99]:
  print(f"{lambdas[i]:10.3f} | {ridge_nonzero[i]:14d} | {lasso_nonzero[i]:14d}")

#Cross-Validation for Optimal λ

pythonrunnable cell

from sklearn.linear_model import RidgeCV, LassoCV

# RidgeCV automatically finds the best alpha via cross-validation
ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 100), cv=5)
ridge_cv.fit(X_scaled, y)
print(f"Best Ridge alpha: {ridge_cv.alpha_:.4f}")

# LassoCV does the same for Lasso
lasso_cv = LassoCV(alphas=np.logspace(-3, 3, 100), cv=5, max_iter=10000)
lasso_cv.fit(X_scaled, y)
print(f"Best Lasso alpha: {lasso_cv.alpha_:.4f}")
print(f"Lasso non-zero features: {np.sum(lasso_cv.coef_ != 0)}")

Watch Out

Regularizing without scaling features first (the silent killer)

Fix: Always run StandardScaler() before Ridge or Lasso. Fit the scaler on training data only, then transform() both train and test sets separately. The regularizer penalizes weight magnitude uniformly — features with large numeric ranges naturally get small weights, which are then penalized less, creating an unfair scale bias. Concrete case: a house-price model with sqft (range 500-5000) and bedrooms (range 1-6). Without scaling, the sqft coefficient is ~0.04 and the bedrooms coefficient is ~10000 — both predicting the same dollar amount per unit change. L2 penalty on the unscaled model punishes bedrooms 60,000,000x harder than sqft, so the model effectively drops bedrooms for no reason other than its units. Scale first, every time.

#Comparison: When to Use Each

	L2 (Ridge)	L1 (Lasso)	ElasticNet
Feature selection	No (keeps all)	Yes (zeros out noise)	Partial (group selection)
Correlated features	Handles well	Picks one, drops rest	Selects groups
Interpretability	Medium (all weights)	High (sparse model)	Medium
Computation	Fast (closed form)	Slower (iterative)	Slower (iterative)
When to use	All features matter	Most features are noise	Features are correlated
Production example

#The Bridge Forward: Which λ?

You now have three regularization knobs (L1 strength, L2 strength, mixing ratio for ElasticNet) — but how do you actually pick the right values? That's the next lesson: Hyperparameter Tuning. The headline result you'll see: log-uniform random search over alpha beats grid search at the same compute, and TPE (Optuna) beats both — but only if you wrap the regularizer in a Pipeline with the scaler so the CV folds don't leak.

And once you've picked λ and trained a tuned, regularized model — the coefficients themselves become an interpretation artifact. Lasso's zeros tell you which features the model relies on. We pick that up in the Model Interpretation lesson.

#Key Takeaways

Overfitting is the problem; regularization is the solution. Without regularization, models memorize training noise via large weights; regularization adds a tax on weight magnitude, forcing simpler solutions that generalize
L2 (Ridge) shrinks all weights toward zero but never to exactly zero. The sphere constraint has no corners; all features are retained; best when all features contain some signal
L1 (Lasso) makes some weights exactly zero. The diamond constraint has corners that align with axes; exact zeros emerge; best for automatic feature selection in high-dimensional sparse problems
ElasticNet combines both penalties. Handles correlated features by selecting groups rather than individuals; the default choice when unsure
Always scale features before regularizing. L1/L2 penalize weight magnitude uniformly; unscaled features with different units get unfair treatment
λ controls the bias-variance tradeoff. Higher λ increases bias but reduces variance; find the sweet spot with cross-validation

#Quick Check

Quick Check1 / 5

You are training a logistic regression model for cancer diagnosis with 5000 genomic features. Most features are likely uninformative SNPs. Which regularization should you use?

You now know how to keep a model from memorizing its training data, and how L1 doubles as automatic feature selection. The same regularization machinery powers weight_decay in every modern neural network optimizer. Next up: Logistic Regression — bending a regularized linear model through a sigmoid to predict probabilities, the workhorse classifier behind spam filters, credit scoring, and click-through-rate models at every major tech company.

Your Reflection

Continue Learning

Discussion

Regularization: L1, L2, and ElasticNet

After this lesson, you will be able to:

Before You Start

#War Story: The Perfect-Fit Model That Was Actually Broken

#The Overfitting Problem

#L2 Regularization (Ridge)

#The Loss Function

#The Gradient

#Geometric Intuition: The Sphere Constraint

#The Unconstrained Solution

#Adding the L2 Constraint (Sphere)

#Where the Ellipses Touch the Sphere

#The Result: Smooth Shrinkage

#L1 Regularization (Lasso)

#The Loss Function

#The Gradient (Subgradient)

#Geometric Intuition: The Diamond Constraint

#The Unconstrained Solution

#The L1 Constraint is a Diamond

#The Loss Contours Hit a Corner

#Sparsity: Some Features Become Exactly Zero

#Concrete example: L1 on 5 features

#ElasticNet: The Best of Both

When to use ElasticNet

#Choosing λ: The Regularization Path

#Cross-Validation for Optimal λ

#Comparison: When to Use Each

#The Bridge Forward: Which λ?

#Key Takeaways

#Quick Check

Deep Dive

Deep Dive