Every "Model B is +1.2% better than Model A" claim is a hypothesis test waiting to happen. Without this lesson, you'll ship models that aren't actually better, and burn $10M on changes that did nothing. P-values and confidence intervals aren't pedantic — they're the difference between "we shipped an improvement" and "we got lucky."
Learning Objectives
After this lesson, you will be able to:
Set up a hypothesis test: state the 'boring explanation' (nothing special is happening) and the 'interesting explanation' (something real is going on)
Understand p-values: the probability of seeing your result by pure luck. A tiny p-value means luck is an unlikely explanation -- but it does not prove anything for certain
Tell apart the two ways a test can be wrong: a false alarm (saying something works when it does not) vs. a missed discovery (saying nothing works when it actually does)
Build and read confidence intervals: a range of values that is likely to contain the true answer
This lesson gives you a superpower: the ability to tell real patterns from lucky coincidences. Every time someone says "this new feature increased sales" or "this drug cures patients," hypothesis testing is how you know if they are right or just fooling themselves.
The idea is simple: assume the boring explanation is true (they are guessing), calculate how unlikely the observed result would be under that assumption, and if it is very unlikely, reject the boring explanation.
Every hypothesis test starts with two competing claims:
Null hypothesis (H0): The "nothing interesting is happening" explanation. Your friend is guessing randomly. The drug has no effect. The new algorithm is no better than the old one.
Alternative hypothesis (H1 or Ha): The "something real is happening" explanation. Your friend has genuine ability. The drug works. The new algorithm is better.
The null hypothesis is the default. You assume it is true unless the evidence is strong enough to reject it. This is like a courtroom: the defendant (null hypothesis) is innocent until proven guilty beyond reasonable doubt.
p < 0.05 -- The result is "statistically significant" at the 5% level. If the null hypothesis were true, you would see a result this extreme less than 5% of the time. Most scientists reject H0 at this threshold.
p < 0.01 -- Stronger evidence against H0. Less than 1% chance of seeing this by luck.
p < 0.001 -- Very strong evidence. Less than 0.1% chance.
p = 0.30 -- Not significant. A 30% chance of seeing this result even if H0 is true. Nothing unusual here.
What Do You Think?
A new drug works in 52% of patients vs 48% for placebo, tested on 100 patients. Is this likely to be statistically significant?
Probably not. With only 100 patients, a 4-percentage-point difference is well within the range of random variation. If you flipped a fair coin 100 times, getting 52 heads would not surprise anyone. You would need either a larger effect size or many more patients (thousands) to detect a small but real difference. This is why clinical trials recruit thousands of participants -- to have enough statistical power to detect real but small effects.
#The Binomial Test: Formalizing the Coke vs Pepsi Problem
For our Coke vs Pepsi example, we use the binomial distribution (from the descriptive statistics lesson). Under the null hypothesis (pure guessing), each trial has p = 0.5 probability of success.
p-value=P(X≥8∣n=10,p=0.5)=k=8∑10(k10)(0.5)10
So the p-value is about 0.055. At the traditional alpha = 0.05 threshold, you would NOT reject the null hypothesis. Your friend's performance is suggestive but not conclusive. If they got 9 out of 10 right, the p-value drops to 0.011 -- now that is significant.
Try it! Open the Python REPL (bottom-right of the screen: click Quick Actions, then Python) and type these lines yourself.
Try it: Visualize the sampling distribution and p-valueInteractive
Loading visualization...
Hypothesis Testing — Drag the Test StatisticInteractive
Loading visualization...
Quick check
A study reports p = 0.03 for a new drug. Which interpretation is correct?
No test is perfect. There are two ways to be wrong:
#Type I Error (False Positive): Convicting an Innocent Person
A Type I error means rejecting the null hypothesis when it is actually true. You conclude the drug works, but it really does not. You ship the new algorithm, but it is actually no better. You convict an innocent defendant.
The probability of a Type I error is called alpha, and it equals your significance threshold (usually 0.05). By setting alpha = 0.05, you are accepting a 5% chance of being fooled by randomness.
#Type II Error (False Negative): Letting a Guilty Person Go Free
A Type II error means failing to reject the null hypothesis when it is actually false. The drug genuinely works, but your test missed it. The new algorithm IS better, but you did not have enough data to detect it. A guilty defendant walks free.
The probability of a Type II error is called beta. The complement (1 - beta) is called statistical power -- the probability of correctly detecting a real effect.
α=P(reject H0∣H0 true)β=P(fail to reject H0∣H0 false)
Medical testing: A Type I error (false positive) on a cancer screening means unnecessary biopsies and anxiety. A Type II error (false negative) means a missed cancer diagnosis. Medical tests are tuned to minimize Type II errors (high sensitivity) even at the cost of more false positives.
Spam filters: A Type I error sends a real email to spam (annoying). A Type II error lets spam into your inbox (also annoying). Email providers balance both based on what users hate more.
ML model comparison: A Type I error means you deploy a "better" model that is actually the same. A Type II error means you stick with the old model when the new one is genuinely better.
Quick check
An A/B test reports p = 0.20 for 'B is better than A.' Which is the right conclusion?
Pretend you ran an A/B test. Group A is the control; group B is the variant. The cell below uses scipy.stats.ttest_ind to compute the t-statistic, p-value, and a 95% confidence interval for the difference of means — exactly the workflow you would use at any growth team.
Loading visualization...
#Confidence Intervals: A Range That Probably Contains the Truth
Instead of just asking "is the effect real?" (hypothesis test), you can ask "how big is the effect?" (confidence interval). A confidence interval gives you a range of plausible values for the true parameter.
A 95% confidence interval does NOT mean "there is a 95% probability that the true value is in this range." That is the most common misinterpretation in all of statistics.
It means: if you repeated this experiment many times and computed a CI each time, 95% of those intervals would contain the true value. Any single interval either contains the truth or it does not -- you just do not know which.
Every tech company runs A/B tests: show version A to half the users and version B to the other half. Hypothesis testing determines whether the difference in click rates, engagement, or revenue is real or noise. If p < 0.05, ship it. If not, the change probably is not worth the complexity.
Is Model B really better than Model A? If Model B gets 91.2% accuracy vs Model A's 90.8%, you need a hypothesis test to know if that 0.4% gap is real. Without it, you might deploy a "better" model that just got lucky on the test set. Cross-validation combined with paired t-tests or bootstrap tests gives you the answer.
When a model achieves 65% accuracy on a binary classification problem, is it actually learning, or could a coin flip do the same? The null hypothesis is "the model is no better than random chance" (50% for binary). A hypothesis test tells you whether the model has learned real patterns. This is especially important for imbalanced datasets where a model predicting "not fraud" 99% of the time gets 99% accuracy without learning anything.
Is this feature actually useful for prediction? Statistical tests (like t-tests and chi-squared tests) help determine whether a feature has a real relationship with the target variable or just a spurious correlation. Dropping noise features improves model generalization.
Tests · Verify that the p-value is computed correctly for known inputs. binomial_pmf(8, 10, 0.5) should be approximately 0.044. compute_p_value(8, 10) should be approximately 0.055.
⚡ Playground:Probability Explorer → — explore sampling distributions and watch the Central Limit Theorem emerge.
Hypothesis testing asks "could this be random noise?" -- You assume the null hypothesis (no effect) is true and calculate the probability of seeing your data under that assumption; if the probability is very low, you reject the null hypothesis
The p-value is NOT the probability that the null hypothesis is true -- It is the probability of seeing data this extreme IF the null hypothesis were true; a subtle but critical distinction
Type I errors (false positives) and Type II errors (false negatives) are in tension -- You can only reduce both by collecting more data; the significance level alpha controls the false positive rate; statistical power (1 - beta) controls the detection rate
Confidence intervals complement hypothesis tests -- Instead of just "is there an effect?", they tell you "how big is the effect?"; wider intervals mean less certainty, narrower intervals mean more data or less variability
In ML, hypothesis testing powers A/B tests, model comparison, and feature selection -- Every time you ask "is this improvement real?" you are doing statistical inference
Your friend gets 8 out of 10 Coke vs Pepsi identifications correct. The p-value is 0.055. At the alpha = 0.05 significance level, what do you conclude?
Next up: Optimization and Gradient Descent. You will learn how machines use gradients to find the bottom of a loss landscape -- the algorithm that powers all of deep learning.