Cross-entropy loss. Mean squared error. L2 regularization. L1 regularization. Every loss function you've ever seen in PyTorch is a re-skin of one idea from this lesson: maximum likelihood estimation. Once you see it, you'll never write loss = nn.CrossEntropyLoss() the same way again — you'll know what it's actually optimizing.
Learning Objectives
After this lesson, you will be able to:
Flip your perspective from probability to likelihood -- the same number p(data | θ), viewed as a function of θ instead of data
Derive Maximum Likelihood Estimators (MLE) from first principles for Bernoulli, categorical, and Gaussian models
See that cross-entropy and MSE -- the two losses that train every modern AI model -- are negative log-likelihoods in disguise
Understand Maximum A Posteriori (MAP) estimation as MLE plus a prior, and recognize L2 / L1 regularization as Gaussian / Laplace priors
Build this --> Build a "Coin Bias Estimator" that takes a sequence of coin flips, computes both the MLE and the MAP estimate (using a Beta prior), and shows how the prior pulls the MAP toward 0.5 when data is sparse but lets it match the MLE when data is plentiful -- watching the prior fade as evidence accumulates.
MLE is the engine that powers every supervised learning algorithm. Once you see that cross-entropy and MSE are both negative log-likelihoods, the entire ML loss-function zoo collapses into a single principle: pick the model that makes the data most probable.
Before we crank the algebra, lock in your gut on the probability-vs-likelihood split.
What Do You Think?
You flip a coin three times and observe HHH. Which statement is TRUE about the likelihood L(θ | HHH) = θ³ as a function of θ?
The likelihood is the same number as the probability — same formula, same arithmetic. The only thing that changes is which variable is fixed and which is free. In probability, θ is fixed (we know the coin is fair) and data varies (we ask "how likely are 3 heads?"). In likelihood, data is fixed (we saw 3 heads) and θ varies (we ask "which θ best explains this?").
Try it! Open the Python REPL (bottom-right of the screen: click Quick Actions, then Python) and type these lines yourself.
Now make the abstraction visible. The block below plots the likelihood surface for a Bernoulli (drag n, k, watch the peak shift), and lets you toggle between MLE-only and MAP-with-prior views. The MLE is wherever the surface peaks.
Loading visualization...
For independent identically distributed (iid) observations, the joint probability factorizes:
Numerical stability: Computers represent floating-point numbers with finite precision. Multiplying 10,000 numbers each ≈ 0.001 gives 10^(-30,000), which is exactly zero in any practical floating-point format. Logging converts the product to a sum: log Π p_i = Σ log p_i. Sums do not underflow.
Mathematical convenience: The log is monotonically increasing, so argmax_θ f(θ) = argmax_θ log f(θ). The maximum stays the same. But the log-likelihood is a SUM, and derivatives of sums are sums of derivatives — much easier to differentiate term-by-term.
This is the canonical example. Master it and the rest follow.
Setup: You flip a (possibly biased) coin n times and observe k heads. The unknown parameter is θ = P(heads). What is θ̂_MLE?
Step 1: Write down the likelihood. For one flip, P(X=1 | θ) = θ and P(X=0 | θ) = 1 − θ. The combined formula P(X=x | θ) = θ^x (1-θ)^(1-x) covers both cases.
For n iid flips with k heads:
L(θ)=θk(1−θ)n−k
Step 2: Take the log. The log-likelihood is:
ℓ(θ)=klogθ+(n−k)log(1−θ)
Step 3: Take the derivative and set to zero.
dθdℓ=θk−1−θn−k=0
Step 4: Solve.
k / θ = (n − k) / (1 − θ)
k (1 − θ) = (n − k) θ
k − kθ = nθ − kθ
k = nθ
θ̂_MLE = k / n
The MLE for a Bernoulli is the empirical success rate: count the heads, divide by total flips. Three heads in four flips → θ̂_MLE = 0.75.
This is the simplest derivation in MLE-land but every other MLE follows the same recipe: write likelihood → take log → take derivative → set to zero → solve.
Let's compute the Bernoulli MLE numerically and watch it converge to the truth as n grows. The playground below draws n Bernoulli samples with θ=0.7, computes the MLE k/n, and plots the log-likelihood — drag the cursor to see how the peak narrows as evidence accumulates.
Use a Lagrange multiplier: form Λ(θ, λ) = Σ_i n_i log θ_i − λ (Σ_i θ_i − 1). Setting ∂Λ/∂θ_i = n_i/θ_i − λ = 0 gives θ_i = n_i/λ. The constraint Σ θ_i = 1 forces λ = n. So:
θ^i,MLE=nni
The MLE for a categorical is what every imbalanced-class baseline uses: predict the most frequent class with probability equal to its empirical frequency.
Quick check
You observe HHHHT (4 heads, 1 tail). The Bernoulli MLE is θ̂ = 0.8. If you instead observed HHHHHHHHTT (8 heads, 2 tails) — same empirical rate — what happens to the MLE and the SHARPNESS of the likelihood peak?
Setup: n iid samples from N(μ, σ²) with both μ and σ² unknown. The likelihood for one sample is the Gaussian PDF:
p(x∣μ,σ2)=2πσ21exp(−2σ2(x−μ)2)
Log-likelihood for n samples:
ℓ(μ,σ2)=−2nlog(2πσ2)−2σ21i=1∑n(xi−μ)2
MLE for μ: take ∂ℓ/∂μ = 0. The first term has no μ. The second term gives (1/σ²) Σ (x_i − μ) = 0, so μ̂_MLE = (1/n) Σ x_i = x̄ (the sample mean).
MLE for σ²: take ∂ℓ/∂σ² = 0. Some calculus gives:
μ^MLE=xˉ,σ^MLE2=n1i=1∑n(xi−xˉ)2
#The Punchline: Cross-Entropy IS Negative Log-Likelihood
This is the moment that ties the entire math track together.
A k-class classifier outputs probabilities p = (p_1, ..., p_k). The true label is class y_true, encoded as a one-hot vector y where y_i = 1 if i = y_true and 0 otherwise.
The categorical likelihood for this one example is:
p(y∣p)=i=1∏kpiyi=pytrue
The negative log-likelihood is therefore:
NLL=−logpytrue=−i=1∑kyilogpi
Cross-entropy IS negative log-likelihood for a categorical model. Every classifier in PyTorch trained with F.cross_entropy(logits, targets) is doing Maximum Likelihood Estimation under a categorical likelihood. Your CNN, your transformer, your Whisper voice model, your CLIP vision-language head — all of them are MLE machines.
When you read about "cross-entropy loss" in a paper, replace it mentally with "negative log-likelihood." It will feel less arbitrary and more inevitable.
#MSE IS Negative Log-Likelihood Under Gaussian Noise
Same trick, regression edition.
Setup: Regression model y = f_θ(x) + ε where the noise ε ~ N(0, σ²) (the standard regression assumption).
The likelihood of observation y given input x and parameters θ is the Gaussian density centered at f_θ(x):
p(y∣x,θ)=2πσ21exp(−2σ2(y−fθ(x))2)
The negative log-likelihood across n iid examples is:
NLL=2nlog(2πσ2)+2σ21i=1∑n(yi−fθ(xi))2
MSE is negative log-likelihood under Gaussian noise. Linear regression, neural network regression heads, CLIP's image-similarity loss, every "regress to a target" objective — all of them assume implicitly that the residuals are Gaussian.
This implicit assumption matters. If your residuals are heavy-tailed (financial returns, latencies, anomaly distances), Gaussian MLE is suboptimal. The right losses for those cases:
Heavy tails / outliers → Huber loss = MLE under a "Gaussian in the middle, Laplace in the tails" noise model.
Symmetric heavy tails → MAE (mean absolute error) = MLE under Laplace noise.
Asymmetric → Quantile loss = MLE for asymmetric Laplace.
Every "robust regression loss" is just MLE under a different noise model.
MLE is great when you have lots of data. With few data, you can overfit: a Bernoulli with k=2 heads in n=2 flips gives θ̂_MLE = 1.0, asserting "this coin only ever lands heads" from two flips. That's clearly silly. Maximum A Posteriori (MAP) estimation fixes this by adding prior information.
θ^MAP=argθmaxp(θ∣D)=argθmaxp(D∣θ)p(θ)
Taking logs:
θ^MAP=argθmax[logp(D∣θ)+logp(θ)]
Compare MLE and MAP side by side on the same Bernoulli. Toggle the prior strength and watch the MAP shift away from 0.75 toward 0.5 — but only when the prior is strong relative to the data.
Loading visualization...
What Do You Think?
You have a Beta(2, 2) prior on θ (a 'weak' prior centred at 0.5). You see 3 heads in 4 flips. The MLE is 0.75. Where does the MAP land?
Run it numerically. The cell below computes both MLE and MAP for the same Bernoulli data, sweeps n, and shows that the gap between them collapses as n grows — that's the prior fading.
This is the classical derivation of ridge regression from Bayesian assumptions. Every line of optimizer = AdamW(params, weight_decay=0.01) is implementing Gaussian-prior MAP estimation. The number 0.01 is the prior precision λ — your assertion that "weights, before seeing data, should be small with standard deviation around 1/√0.01 = 10."
The Laplace distribution is shaped like a tent (sharper peak, heavier tails than Gaussian). Its negative log-density is |θ|/b for a scalar.
p(θ)=Laplace(θ∣0,b)⟹−logp(θ)=b1∥θ∥1+const
LASSO regression, sparse coding, and L1-regularized neural networks all implement Laplace-prior MAP estimation.
Quick check
You add `weight_decay=0.01` to an AdamW optimizer for a model with 7 billion parameters. From a MAP perspective, what implicit assumption are you encoding?
Define the score function as the gradient of the log-likelihood with respect to θ:
s(θ;x)=∇θlogp(x∣θ)
At the MLE, the sum of scores across all data points equals zero. That IS the equation dℓ/dθ = 0 we solved for the Bernoulli case. The score is the engine MLE uses to find its solution.
The score also has a deeper role: it is the "data direction" that diffusion models learn. Score matching trains a model to predict ∇_x log p(x) (the score with respect to data x rather than parameters θ). The math is the same; only the variable being differentiated changes.
The Fisher information captures how much information one observation carries about θ. Formally:
I(θ)=Ex[s(θ;x)s(θ;x)⊤]=−Ex[∇θ2logp(x∣θ)]
Fisher information shows up everywhere in advanced ML:
Cramér-Rao bound: Var(θ̂) ≥ I(θ)^(-1) for any unbiased estimator. Fisher lower-bounds variance — you cannot estimate better than the data allows.
Asymptotic efficiency of MLE: as n → ∞, the MLE achieves the Cramér-Rao bound. MLE is the best estimator you can have for large datasets.
Natural gradient: the update θ_{t+1} = θ_t − η F^(-1) ∇L preconditions the gradient by inverse Fisher. Geometrically, this is the steepest-descent direction in the parameter manifold's natural metric, not Euclidean parameter space. Used in K-FAC, Adam-W's interpretation, natural-gradient methods.
What Do You Think?
The Fisher information for a Bernoulli is I(θ) = 1 / [θ(1 − θ)]. At which θ is the Fisher information LARGEST — i.e., which coin gives you the MOST information per flip?
When you write loss = F.cross_entropy(logits, targets) and call loss.backward(), you are computing the negative log-likelihood of the data under your model's predicted categorical distribution, then backpropagating its gradient. Every classifier in modern ML — image classification, language modeling, speech recognition — is doing categorical MLE. This is not an arbitrary loss choice; it is the maximum-likelihood objective falling out of the categorical assumption.
optimizer = AdamW(model.parameters(), lr=3e-4, weight_decay=0.01) implements MAP estimation under a zero-mean Gaussian prior with precision 0.01. Without weight decay, large models overfit catastrophically. With weight decay, the prior keeps weights small unless data forces them to be large — which is the entire point of regularization. Every transformer training run, including GPT-4 and Claude, sets weight_decay through this principle.
In RLHF, given preference pairs (chosen, rejected), the reward model fits P(chosen ≻ rejected) = σ(r(chosen) − r(rejected)) (Bradley-Terry). Training the reward model is MLE under this preference likelihood. The standard "preference loss" −log σ(r(chosen) − r(rejected)) is just the negative log-likelihood. Same recipe, different model.
Diffusion training does not maximize likelihood directly — the data likelihood under a diffusion model is intractable. Instead, the network is trained to predict ∇_x log p_t(x) (the score function with respect to noisy data at time t). This is the same ∇ log p from MLE, but differentiated w.r.t. data rather than parameters. Score matching is MLE's tractable cousin for high-dimensional density modeling.
MAP gives you a single best θ. Bayesian deep learning approximates the FULL posterior p(θ | D), capturing weight uncertainty. Practical methods (Monte Carlo dropout, deep ensembles, Laplace approximation around the MAP) all start from MLE/MAP and try to characterize how uncertain that estimate is. The same likelihood machinery scales to full Bayesian inference; only the question changes from "best θ" to "distribution over θ."
Probability & Bayes gave you the formula p(θ|D) ∝ p(D|θ) p(θ). MAP is what happens when you take the argmax of that posterior.
Random Variables, Expectation & Variance gave you E[X] and Var(X). The Cramér-Rao bound gives you a lower bound on Var(θ̂) in terms of Fisher information.
Descriptive Statistics showed you sample variance with the Bessel correction. Now you know WHY that correction exists: MLE for variance is biased; (n−1) makes it unbiased.
Multivariate Gaussian introduced the Gaussian density and the change-of-variables formula. Gaussian MLE for both μ and Σ pops out of multivariate calculus.
Information Theory introduced entropy, cross-entropy, and KL divergence. Cross-entropy is exactly negative log-likelihood for categorical models — same number, two names from two communities (information theorists and statisticians).
Statistical Inference gave you p-values and confidence intervals. The asymptotic normality of MLE θ̂ ~ N(θ_true, I(θ)^(-1)) gives you the standard error and CIs for any MLE.
Calculus & Derivatives gave you the gradient. Score function = gradient of log-likelihood. Fisher information = negative expected Hessian. Natural gradient uses both.
Matrix Calculus & Backprop gave you the chain rule for vectors. The end-to-end backprop you derived for softmax + cross-entropy is the gradient of the categorical NLL — every modern ML training loop is gradient ascent on the log-likelihood.
Once you see MLE under all the losses, you stop memorizing loss functions. You start deriving them from probabilistic assumptions about your data.
Next up: Markov Chains and MDPs — the math foundation of reinforcement learning, diffusion forward processes, autoregressive sampling, and PageRank. You will learn what a stationary distribution is, how Bellman equations work, and why "Markov" shows up everywhere in modern ML.