Tinder swipes. Spam folders. Fraud alerts on your credit card. Cancer screenings that flag a scan for a doctor's second look. All of these answer the same yes/no question — but instead of a hard "yes" or "no," they want a probability: "97% chance this is spam," "12% chance this transaction is fraud." Plain linear regression can't do that — it'll happily output -0.3 or 1.7, which makes no sense as a probability. By the end of this lesson, you'll build a classifier that turns hours-studied into a "probability of passing the exam" — and you'll see why the same little S-curve trick is sitting inside every spam filter, every credit score, and every output neuron of every AI chatbot you have ever used.
Learning Objectives
After this lesson, you will be able to:
Understand why you cannot use a straight line for yes/no predictions and how the S-shaped sigmoid curve fixes it
Build the logistic regression model step by step and interpret its output as a probability (like '73% chance of spam')
Understand binary cross-entropy — the loss function that tells the model how wrong its probability predictions are
Visualize the decision boundary — the line where the model switches from 'yes' to 'no'
Know when to choose logistic regression over a decision tree or SVM — and what 'linearly separable' actually means in practice
Build this --> Build a spam classifier: take 100 emails (50 spam, 50 real), extract features like word count, number of links, and presence of suspicious keywords, train logistic regression, and watch it sort new emails into spam vs. inbox with a confidence percentage
Don't worry if "logistic regression" sounds complicated — it is really just linear regression with one extra step: squishing the output through an S-curve so it becomes a probability between 0% and 100%. That is the whole trick!
You start with a set of features for each example -- hours studied, previous test scores, attendance rate. Each example also has a binary label: pass or fail, spam or not spam, malignant or benign. The goal is to learn a function that maps features to the probability of the positive class.
Just like linear regression, compute a weighted sum of the input features: z = w1x1 + w2x2 + ... + b. This produces a raw score that can be any real number -- positive, negative, or zero. The weights determine how much each feature contributes to the decision.
The raw score z is passed through the sigmoid function: sigma(z) = 1 / (1 + e^(-z)). This S-shaped curve maps any real number to a value between 0 and 1. Large positive z maps to near 1, large negative z maps to near 0, and z = 0 maps to exactly 0.5. The output is now a valid probability.
Compare the predicted probability to a threshold (typically 0.5). If P(y=1|x) > 0.5, predict the positive class. If P(y=1|x) <= 0.5, predict the negative class. The threshold is a business decision -- for cancer screening, you might lower it to 0.3 to catch more true positives at the cost of more false alarms.
The model outputs both a probability and a class label. During training, binary cross-entropy loss measures how far the predicted probabilities are from the true labels, and gradient descent adjusts the weights to minimize this loss. The decision boundary -- where P = 0.5 -- is always a straight line (hyperplane) in feature space.
Sigmoid Explorer — Drag z, Adjust Threshold, See P(y=1) ChangeInteractive
Loading visualization...
σ(z)=1+e−z1
Try it! Open the Python REPL and type these lines yourself. See the sigmoid in action: import numpy as np; sigmoid = lambda z: 1/(1+np.exp(-z)); print(f"sigmoid(-5)={sigmoid(-5):.5f}, sigmoid(0)={sigmoid(0):.1f}, sigmoid(5)={sigmoid(5):.5f}") — watch how it squishes any number into the 0-to-1 range!
Key properties of the sigmoid:
Range: Always between 0 and 1 (perfect for probabilities)
The model outputs a probability, not a hard classification. To get a class prediction, threshold at 0.5 (or any threshold appropriate for your problem):
P(y=1|x) > 0.5 --> predict class 1
P(y=1|x) <= 0.5 --> predict class 0
What Do You Think?
For logistic regression with weights w = [2, -3] and bias b = 1, what is the predicted class for input x = [1, 1]?
With w = [2, -3] and x = [1, 1]: z = 2(1) + (-3)(1) + 1 = 0. sigmoid(0) = 0.5. The point is exactly on the decision boundary. In practice, most implementations assign this to class 0 (since P is not strictly greater than 0.5), but the key insight is that z = 0 defines the decision boundary.
#Worked Example: Predicting Exam Pass from Hours Studied
Say you've already trained a model on past students and ended up with weights w = 1.2 and bias b = -6. Now plug in real students:
Student
Hours (x)
z = 1.2·x − 6
σ(z) = P(pass)
Predict
Ana
2
−3.6
0.027
FAIL (3%)
Ben
4
−1.2
0.231
FAIL (23%)
Cara
5
0.0
0.500
borderline
Dev
6
1.2
0.769
PASS (77%)
Eli
8
3.6
0.973
PASS (97%)
Two things to notice. First, the decision boundary is at hours = 5 — solving z = wx + b = 0 gives x = −b/w, and with w = 1.2 and b = −6 that's 6/1.2 = 5. That's the input value where the model flips from "fail" to "pass." Second, the probabilities are not linear — Ana and Ben both fail but Ana fails with much more confidence (3% vs 23%), and Dev and Eli both pass but Eli passes with overwhelming confidence (97% vs 77%). The sigmoid does this naturally: far from the boundary, the model is very confident; close to the boundary, it hedges. That's calibrated uncertainty, and it's the entire reason we wrap a sigmoid around the line in the first place.
Try it: Drag the weights and bias to move the decision boundaryInteractive
Loading visualization...
Try this: Observe how the decision boundary is a straight line in 2D (a hyperplane in higher dimensions). Drag the weights and bias to see how the boundary moves. Notice that points on one side are classified as class 0, and points on the other side as class 1. The boundary is where P(y=1) = 0.5, which corresponds to w^T x + b = 0.
Interactive Lab
Move training points around and watch the logistic-regression boundary re-fit. Compare it with KNN and SVM on the same data — same problem, very different shapes.
The decision boundary of logistic regression is always linear (a straight line/plane/hyperplane). This means logistic regression can only separate classes that are linearly separable -- you can draw a straight line between them. For non-linear boundaries, you need kernel methods, polynomial features, or neural networks.
Let's actually train one. Below, we generate two overlapping blobs, fit LogisticRegression, draw the learned boundary, and check accuracy on a held-out test set.
Loading visualization...
What Do You Think?
A logistic regression model on customer-churn data learns weight w_age = +0.20 for age (after standardizing the feature). What does the coefficient mean?
For more than two classes, logistic regression generalizes to softmax regression (also called multinomial logistic regression):
P(y=k∣x)=∑j=1Kewj⊤x+bjewk⊤x+bk
Quick check
In softmax regression with 4 classes, you compute scores z = [2.0, 1.0, 0.1, -1.0]. Without doing the full computation, which class wins, and roughly why?
In scikit-learn, LogisticRegression applies L2 regularization by default (C=1.0). This is different from LinearRegression, which has no regularization by default.
Tests · Verify the decision boundary is near 5 hours. Check that P(pass|1 hour) < 0.1 and P(pass|10 hours) > 0.9. Modify the learning rate and observe convergence speed.
The sigmoid squeezes any number into a probability. By wrapping a linear function in the sigmoid, logistic regression outputs calibrated probabilities between 0 and 1 instead of unbounded predictions
Cross-entropy is the correct loss for classification. It creates a convex loss surface (guaranteed single global minimum) and catastrophically penalizes confident wrong predictions, driving well-calibrated probabilities
The decision boundary is always linear. Logistic regression can only separate classes with a straight line/hyperplane; for non-linear boundaries, you need polynomial features, kernels, or neural networks
Softmax generalizes to multiple classes. For K > 2 classes, softmax computes a probability distribution over all classes, with binary logistic regression as a special case
Accuracy is misleading for imbalanced classes. With 99% negative examples, always predicting negative gives 99% accuracy but catches nothing; use precision, recall, F1, or AUC instead
Why is cross-entropy used instead of MSE for logistic regression?
Key Terms8 terms
S-shaped curve sigma(z) = 1 / (1 + e^(-z)) that squashes any real number into the range (0, 1), turning a raw score into a probability.
The raw linear score z = w^T x + b, also equal to log(p / (1 - p)). Moving one unit in a feature changes the log-odds by that feature's weight.
The set of inputs where the predicted probability equals the threshold (typically 0.5). For logistic regression this is always a linear hyperplane.
The loss function used for logistic regression: -[y log(p) + (1-y) log(1-p)]. It is convex in the model's weights and heavily penalizes confident wrong predictions.
Multi-class generalization of sigmoid. Takes a vector of scores and normalizes them into a probability distribution over K classes via exp() and division by the sum.
When one class is far more common than others. Accuracy becomes misleading and models tend to ignore the minority class; mitigated with class weights, resampling, or threshold tuning.
How closely predicted probabilities match observed frequencies. A well-calibrated model that says '70% chance' is right about 70% of the time.
Penalty added to the loss (sum of |w| for L1, sum of w^2 for L2) to shrink weights and prevent overfitting. Scikit-learn's C parameter is the inverse of the regularization strength.
Where This Matters
Google
Spam Detection at Gmail
Gmail's classic spam pipeline (still a core signal even alongside deep models) uses logistic regression over features like sender reputation, link patterns, and user interaction signals to output a spam probability.
↑
Blocks ~100M spam messages per day with >99.9% accuracy
FICO
Credit Scoring (FICO / Banks)
FICO and most consumer lenders use logistic regression for credit risk scoring because regulators (ECOA, Fair Lending Act) require that every denial be explainable via concrete factor weights.
↑
Governs >$4T in annual US consumer lending decisions
Meta
Ad Click Prediction at Meta / Google
Before a click-through-rate model upgrades to deep networks, the production baseline is large-scale logistic regression (often trained on trillions of feature crosses) because it serves predictions in microseconds at global scale.
↑
Serves billions of ad impressions per day in under 100ms
Interview Practice
You now know how to classify with probabilities. But logistic regression draws straight lines -- what if the boundary between classes is complex? Next up: Decision Trees, which ask a series of yes/no questions to carve up feature space into irregular regions.