Linear & Polynomial Regression

Zillow's "Zestimate" looks at a house in your neighborhood and tells you its price within a few percent. Strava predicts your finishing time from your training pace. Your fitness app tells you how many calories you burned from heart rate and weight. All three are doing the same thing under the hood: drawing the best line through past data and using it to predict the future. By the end of this lesson, you'll build a regression model that predicts house prices from square footage — and you'll understand exactly why each coefficient ("each extra square foot adds $147 to the price") is the number it is. No magic. Just one line, two parameters, and a tiny bit of arithmetic.

Don't worry if the math notation looks scary — linear regression is literally just drawing the best straight line through your data points. If you can plot points on graph paper and eyeball a line through them, you already get the core idea!

#The Simplest Prediction Machine

#The Model

A linear regression model predicts a continuous output as a weighted sum of input features:

In compact matrix notation for all n training examples at once:

Try it! Open the Python REPL and type these lines yourself. Fit a line in 3 lines of code: import numpy as np; x = np.array([1,2,3,4,5]); y = np.array([2,4,5,4,5]); coeffs = np.polyfit(x, y, 1); print(f"Slope: {coeffs[0]:.2f}, Intercept: {coeffs[1]:.2f}") — you just ran linear regression!

Where:

• X is the n x d data matrix (n samples, d features)
• w is the d-dimensional weight vector
• b is the bias (intercept) term
• y-hat is the vector of predictions

#Worked Example: 3 Houses and a Best-Fit Line

Let's make this concrete. You have 3 houses:

• Σx = 4500, Σy = 1,250,000, Σxy = 1000·300K + 1500·400K + 2000·550K = 2,000,000,000, Σx² = 1,000,000 + 2,250,000 + 4,000,000 = 7,250,000, n = 3
• w = (3 · 2,000,000,000 − 4500 · 1,250,000) / (3 · 7,250,000 − 4500²) = 375,000,000 / 1,500,000 = 250
• b = (Σy − w · Σx) / n = (1,250,000 − 250 · 4500) / 3 = 125,000 / 3 ≈ 41,667

• Slope w = 250 -- each additional square foot adds $250 to the price.
• Intercept b = 41,667 -- a "0 sqft house" would (in theory) cost $41.7K. This is the value of the lot/permits/foundation independent of size. The intercept rarely has a real-world meaning -- it's just the y-axis crossing point.

#Explore: Fitting a Line

#The Loss Function: Mean Squared Error

Squaring serves two purposes: (1) it makes all errors positive without the non-differentiable absolute value, and (2) it penalizes large errors quadratically, making the model focus on reducing big mistakes. A prediction off by 10 contributes 100 to the loss, while a prediction off by 1 contributes only 1. Both reasons matter, so "both" is the correct answer.

#The Closed-Form Solution: Normal Equation

This is beautiful: one matrix equation, computed in one step, gives you the globally optimal weights. No learning rate, no epochs, no hyperparameter tuning.

#When Does the Closed-Form Break Down?

The normal equation requires inverting the d x d matrix X^T X. This is O(d^3) in computation and O(d^2) in memory. For d = 100 features, this is instant. For d = 1,000,000 features (common in NLP), it is computationally infeasible. In those cases, we fall back to gradient descent, which scales much better with dimensionality.

#Gradient Descent for Linear Regression

When the closed-form is too expensive, we optimize iteratively. The gradient of MSE with respect to weights is:

• There is exactly one global minimum (no local minima)
• Gradient descent is guaranteed to find it (with appropriate learning rate)
• The loss landscape is the friendliest possible terrain for optimization

#Regularization: Taming Overfitting

When you have many features, the model can assign large weights to fit noise. Regularization adds a penalty for large weights:

#Ridge Regression (L2 Regularization)

#Lasso Regression (L1 Regularization)

#Polynomial Regression: Fitting Curves

What if the relationship is not linear? You can fit curves by creating polynomial features:

A degree-20 polynomial has 21 free parameters for 25 data points. It will almost certainly overfit, producing a wildly oscillating curve that hits most training points but generalizes terribly. This is the bias-variance tradeoff in action: too much flexibility lets the model chase noise.

#Try It Yourself

#When to Use Linear Regression vs. Alternatives

#Key Takeaways

• Linear regression is the foundation of ML. It predicts a continuous output as a weighted sum of features, and understanding it deeply gives you 80% of what makes all ML work
• MSE penalizes large errors quadratically. Squaring errors makes them all positive, differentiable everywhere, and disproportionately punishes big mistakes over small ones
• The Normal Equation gives exact weights in one step. Unlike iterative methods, it solves for the global optimum directly, but becomes infeasible for datasets with millions of features
• Regularization prevents overfitting. Ridge (L2) shrinks all weights toward zero while Lasso (L1) can push weights to exactly zero, performing automatic feature selection
• Polynomial regression is still linear in the parameters. Creating features like x-squared and x-cubed lets a linear model fit curves, but high-degree polynomials risk severe overfitting

#Quick Check