Bayes' theorem from 1763 is what makes spam filters work, what doctors use to update diagnoses, and what GPT-4 implicitly uses every time it picks the next word. It's three lines of arithmetic with one idea: "how do I update my belief when I see new evidence?" That's all probability theory is — a calculus for updating your guesses as new data arrives.
Learning Objectives
After this lesson, you will be able to:
Understand conditional probability as narrowing your focus -- 'given that X already happened, how likely is Y?'
Apply Bayes' theorem to flip a probability around: go from 'how likely is the evidence if I am right' to 'how likely am I right given the evidence'
Spot the base rate trap: understand why a '99% accurate' test can still be wrong most of the time when the thing you are testing for is rare
Chain multiple evidence updates together using Bayes' theorem sequentially, as a spam filter does with each suspicious word it encounters
Probability is how smart people deal with uncertainty. You already use it every day -- checking whether to carry an umbrella, deciding if a text is real or spam, guessing if your crush likes you back. This lesson just turns your gut feelings into precise numbers.
Probability is the language of uncertainty, and uncertainty is everywhere in AI. A classifier does not say "this is a cat" -- it says "there is a 94% probability this is a cat." Understanding probability is not optional in machine learning. It is foundational.
A probability P(A) is a number between 0 and 1 that measures how likely event A is to occur. P(A) = 0 means impossible. P(A) = 1 means certain. P(A) = 0.5 means a coin flip.
The sample space is the set of all possible outcomes. For a coin flip: heads or tails. For a die roll: 1 through 6. For an image classifier: cat, dog, bird, fish, and so on.
Two key rules that every probability must satisfy:
All probabilities are between 0 and 1
The probabilities of all possible outcomes sum to 1
F.softmax(logits, dim=-1) is exactly this — it normalizes a vector of real numbers into a probability distribution. The output of every classifier in the world ends with this operation.
P(rain | clouds) ≈ 0.40 in your city. What can you say about P(clouds | rain)?
This is the most important formula in machine learning. It tells you how to reverse a conditional probability -- going from "what is the probability of evidence given the hypothesis" to "what is the probability of the hypothesis given the evidence."
P(H∣E)=P(E)P(E∣H)⋅P(H)
Where:
P(H|E) = posterior -- what we want: the probability of the hypothesis after seeing evidence
P(E|H) = likelihood -- how probable is this evidence if the hypothesis is true?
P(H) = prior -- what we believed before seeing any evidence
P(E) = evidence -- the overall probability of seeing this evidence (normalizing constant)
This is the single most counterintuitive result in probability. Understanding it deeply will change how you think about classification in ML.
What Do You Think?
A medical test has 99% sensitivity (detects disease 99% of the time when present) and a 5% false-positive rate (reports positive 5% of the time when the patient is healthy). The disease affects 1% of the population. You test positive. What is the probability you actually have the disease?
Let us work through this step by step. Prepare to have your intuition shattered.
Before taking the test, there is only a 1% chance you have the disease. This is the base rate -- and it turns out to be the most important number in the calculation.
Only a 16.7% chance you are actually sick. Despite a 99% accurate test!
Why? Picture 10,000 people being tested. 100 have the disease (1%). Of those 100, the test catches 99 (true positives). But of the 9,900 healthy people, 5% test falsely positive: 495 false positives. So out of 594 total positive results (99 + 495), only 99 are truly sick. That is 99/594 = 16.7%.
The base rate matters far more than the test accuracy. This is the base rate fallacy.
The base rate fallacy: out of 10,000 people tested, false positives (495) vastly outnumber true positives (99). Adjust the prior in the visualization below to see how prevalence changes the outcome.
The visualization above shows exactly why the base rate matters so much. With 1% prevalence in a population of 10,000, only 100 people are actually sick -- but 495 healthy people test falsely positive. The true positives are drowned out by false positives from the much larger healthy group. Try adjusting the prior probability to see how the balance shifts.
Reading the math is one thing — running it is another. The cell below codes the disease-test calculation from scratch and lets you sweep prevalence to see the posterior collapse from "I am almost certainly sick" to "almost certainly a false alarm." A Monty Hall solver follows as a second snippet.
Flip the coin and watch your posterior probability update in real-time as evidence accumulates:
Try it: Flip coins and watch beliefs update with each flipInteractive
Loading visualization...
Try this: Start with a prior of 0.5 (maximum uncertainty). Flip the coin several times and watch how the posterior converges. Then reset and start with a strong prior of 0.9 -- notice how it takes more evidence to shift a confident prior.
Try this: Adjust the parameters and watch how the distribution shape changes. A normal distribution with small variance concentrates probability near the mean. A uniform distribution spreads probability evenly.
Tests · Verify the posterior after 'lottery' is above 0.95 and after 'meeting' it drops significantly. Try adding your own word evidence.
An Essay towards solving a Problem in the Doctrine of Chances
Thomas Bayes (posthumous) (1763)
The original paper by Bayes, published after his death by Richard Price. Remarkably accessible for an 18th-century mathematical paper. Introduces what we now call Bayes' theorem through a thought experiment involving billiard balls.
⚡ Playground:Probability Explorer → — adjust priors and likelihoods and watch posterior beliefs update live.
Bayes' theorem reverses conditional probabilities. It converts P(evidence|hypothesis) into P(hypothesis|evidence), letting you update beliefs as new data arrives
The base rate fallacy is critical for ML. Even a 99% accurate test yields only 16.7% true positive rate when the condition is rare (1% prevalence), because false positives from the large healthy population overwhelm true positives
Priors encode what you already know. Your initial belief (prior) gets updated by evidence (likelihood) to produce an updated belief (posterior); with enough evidence, the prior becomes irrelevant
P(A|B) is not the same as P(B|A). This asymmetry is the entire reason Bayes' theorem exists and is a common source of reasoning errors in both medicine and ML
Bayesian thinking pervades ML. From spam filters and Bayesian optimization to regularization (priors on weights) and generative models, Bayes' theorem provides the mathematical framework for reasoning under uncertainty
In the medical test example, 1% disease prevalence and 99% test accuracy yield only a 16.7% chance of disease given a positive test. What would MOST increase the posterior probability?
Next up: Descriptive Statistics and Distributions -- the language we use to summarize data and the named distributions (normal, Bernoulli, binomial, Poisson) that show up everywhere in ML. You will learn variance, the 68-95-99.7 rule, and the Central Limit Theorem that ties this all together.