One question — "how surprised should I be?" — birthed the field that compresses JPEGs, error-corrects 5G signals, and trains every neural classifier. Cross-entropy loss is the king of ML losses. KL divergence is the soul of variational autoencoders and RLHF. Both are children of one equation Shannon wrote in 1948. Information theory is short, deep, and built on coin flips.
Learning Objectives
After this lesson, you will be able to:
Measure how surprising an event is using a simple formula: rare events carry more information, common events carry less
Calculate entropy -- the average surprise of a random process -- and understand why a coin flip is maximally surprising while a rigged game is boring
See why cross-entropy (a specific way to measure average surprise) is the go-to error formula for AI classifiers, and how it measures the gap between what the model predicts and what actually happens
Understand KL divergence as the 'extra surprise' from using the wrong distribution, and explain why minimizing cross-entropy and minimizing KL divergence are equivalent during training
This lesson connects everything. Probability told you how likely things are. This lesson tells you how much you LEARN when something actually happens. It is the mathematical bridge between uncertainty and knowledge -- and it is the reason AI classifiers use the error formula they do.
Claude Shannon, working at Bell Labs in 1948, asked a deceptively simple question: "How do you measure information?" His answer created an entire field and laid the mathematical foundation for the digital age -- and, as it turns out, for the error-measuring formula used in every modern AI classifier.
Try it! Open the Python REPL (bottom-right of the screen: click Quick Actions, then Python) and type these lines yourself.
The information content (or "surprise" or "self-information") of an event is:
I(x)=−log2P(x)
Why the logarithm? Two reasons:
Additivity: Learning two independent facts should give you the sum of each fact's information. Since P(A and B) = P(A) x P(B) for independent events, log turns multiplication into addition: I(A and B) = I(A) + I(B).
Scale: The log gives us natural units (bits when using log base 2, nats when using natural log). One bit is the information gained from a fair coin flip.
What Do You Think?
A weather forecaster says 'It will be sunny tomorrow' in Phoenix, Arizona (sunny 300 days/year). The same prediction in Seattle (sunny 150 days/year). Which prediction carries more information?
The Seattle prediction carries more information. P(sunny in Phoenix) is about 0.82, giving I = 0.29 bits. P(sunny in Seattle) is about 0.41, giving I = 1.29 bits. A sunny day in Seattle is more surprising (less expected), so learning about it gives you more information.
Quick check
A 4-class classifier outputs probabilities [0.97, 0.01, 0.01, 0.01]. The true label turns out to be class 1. What is the information content (in bits) of this observation under the model's distribution?
Entropy measures the average amount of surprise (information) you get from a probability distribution. It answers: "On average, how surprised will I be by the next event?"
The maximum possible entropy for 4 classes is $\log_2 4 = 2.0$ bits (uniform distribution). This classifier is fairly confident — its entropy of 1.245 bits is below the maximum, reflecting that class 0 dominates the probability.
Maximum entropy occurs when all outcomes are equally likely. A fair coin (1 bit) has more entropy than a biased coin. A fair 6-sided die (2.58 bits) has more entropy than a loaded die.
Minimum entropy (zero) occurs when one outcome has probability 1. Zero uncertainty -- you know exactly what will happen.
Entropy measures uncertainty, not randomness. A distribution with high entropy means the next outcome is hard to predict.
Try it: Adjust the distribution and watch entropy changeInteractive
Loading visualization...
Try this: Adjust the parameters to make the distribution more uniform (spread out) and watch entropy increase. Then make one outcome very likely and watch entropy drop toward zero. Try to create the maximum entropy distribution for 4 outcomes -- it should be uniform [0.25, 0.25, 0.25, 0.25] with entropy = 2.0 bits.
What Do You Think?
Which of these three distributions over 4 outcomes has the HIGHEST entropy?
#End-to-End: How Information Flows Through a Channel
Shannon's theory describes the complete journey of information from sender to receiver. This process is the foundation of all digital communication -- and it maps directly to how ML models encode and decode data.
Information starts as a message -- a sequence of symbols drawn from a probability distribution. The entropy H(X) of this source tells you the minimum number of bits needed per symbol. A highly predictable source (like English text, where "e" is common and "z" is rare) has lower entropy and can be compressed more. A completely random source has maximum entropy and cannot be compressed at all.
The encoder compresses the message to remove redundancy. Shannon proved that you can compress down to the entropy (H bits per symbol) but no further without losing information. Huffman coding, arithmetic coding, and zip all approach this theoretical limit. In ML, the encoder is analogous to the feature extraction layers of a neural network -- stripping away noise and keeping the essential signal.
The encoded bits travel through a noisy channel -- a wire, wireless signal, or network. The channel can flip bits, add noise, or lose data. Shannon's channel capacity theorem gives the maximum rate at which information can be transmitted reliably over a noisy channel. This is directly analogous to the information bottleneck in neural networks: each layer is a noisy channel through which data representations must pass.
The decoder reverses the encoding, reconstructing the original message from the noisy received bits. Error-correcting codes add strategic redundancy so that bit flips can be detected and corrected. In ML, the decoder is analogous to the output layers -- the classifier or generator that reconstructs meaning from compressed representations. Cross-entropy loss measures how well your decoder (model) reconstructs the true distribution.
The received message is the decoder's best reconstruction of the original. The cross-entropy between the original source distribution and the decoded output measures how much information was lost or distorted. A perfect channel with a perfect decoder achieves cross-entropy equal to the source entropy -- zero KL divergence, zero wasted bits. Every classification model you train is trying to be a perfect decoder.
Now here is where information theory connects directly to ML. Cross-entropy measures the average surprise when you use one distribution (your model's predictions Q) to encode events that actually come from another distribution (the true distribution P).
H(P,Q)=−x∑P(x)logbQ(x)
This is the most common loss function in classification. When you train a neural network with cross-entropy loss, you are telling the model: "Be as unsurprised as possible by the true labels." Minimizing cross-entropy forces the model's predictions Q to match the true distribution P.
Drag the two distributions P and Q below to see how entropy, cross-entropy, and KL divergence respond live. Pay attention to the moment Q matches P — KL drops to exactly zero, and cross-entropy collapses onto entropy.
Loading visualization...
Quick check
The true distribution is P = [0.5, 0.5]. Your model predicts Q = [0.9, 0.1]. Is the cross-entropy H(P, Q) larger or smaller than H(Q, P)?
In practice, for a single training example with true class k, the true distribution P is a one-hot vector (all zeros except a 1 at position k). The cross-entropy simplifies to:
KL divergence (Kullback-Leibler divergence) measures how different two distributions are. It is the "extra" surprise from using Q instead of P:
DKL(P∥Q)=x∑P(x)logQ(x)P(x)
The crucial relationship: Cross-Entropy = Entropy + KL Divergence.
H(P,Q)=H(P)+DKL(P∥Q)
Since entropy H(P) is fixed (it depends only on the true distribution, not your model), minimizing cross-entropy is exactly the same as minimizing KL divergence. Training pushes your model's distribution as close to the truth as possible.
RLHF's full loss is r(x, y) − β · KL(π_RL || π_SFT). The KL term is exactly this lesson's KL divergence, used as a leash that prevents the fine-tuned model from drifting too far from the supervised baseline. The β coefficient (typically 0.1) controls how tight the leash is.
Cross-Entropy and KL DivergenceInteractive
Loading visualization...
#Jensen's Inequality: The Tool That Proves Everything
Almost every "≥ 0" claim in information theory and variational inference is a corollary of one inequality. Jensen's inequality says: for a convex function f and any random variable X,
f(E[X])≤E[f(X)]
Why this matters: the log function is concave, so applying Jensen's inequality with f = -log gives E[-log X] ≥ -log E[X]. This single fact proves:
KL ≥ 0: D_KL(P‖Q) = E_P[-log(Q/P)] ≥ -log E_P[Q/P] = -log 1 = 0, with equality iff Q = P everywhere. KL divergence is non-negative and zero only when distributions match.
Cross-entropy ≥ entropy: H(P, Q) = H(P) + D_KL(P‖Q) ≥ H(P). The minimum-possible loss when training a classifier is the entropy of the true distribution -- you cannot do better.
The ELBO (next section) -- a lower bound on the log-likelihood derived directly from Jensen.
Entropy measures how uncertain a variable is. Conditional entropy measures how uncertain X remains after you know Y:
H(X∣Y)=−x,y∑P(x,y)logP(x∣y)=Ex,y[−logP(x∣y)]
Mutual information is the reduction in uncertainty about X when you observe Y:
I(X;Y)=H(X)−H(X∣Y)=H(Y)−H(Y∣X)=H(X)+H(Y)−H(X,Y)
Equivalently, I(X; Y) = D_KL(P(x, y) ‖ P(x)P(y)) — mutual information is the KL divergence between the joint distribution and the product of marginals. If X and Y are independent, the joint factors into the product, so KL = 0 and MI = 0. The further the joint deviates from independence, the more information they share.
InfoNCE bridge. Contrastive learning objectives like InfoNCE (CLIP, SimCLR, MoCo) maximize a lower bound on mutual information. CLIP trains image-text pairs so that I(image; text) is high — this is what "the image and its caption are about the same content" means in information-theoretic terms. The training objective penalizes the model when its joint distribution over (image, text) cannot be distinguished from the product of marginals.
The Evidence Lower BOund (ELBO) is the punchline that ties information theory to variational autoencoders, Bayesian deep learning, and modern diffusion models. It answers: given a probabilistic model with latent variables z, how do I train its parameters when log p(x) is intractable to compute?
Suppose you want to maximize log p(x) (the log-likelihood of the data under your model), but p(x) = ∫ p(x, z) dz requires an integral you cannot evaluate. Introduce an auxiliary distribution q(z) (the variational posterior), and apply Jensen to the log:
That is the VAE objective. Reconstruction term + KL regularizer = ELBO. Maximizing ELBO is what every VAE in production does. Diffusion models extend this to a chain of latent variables x_T → x_ → ... → x_0 (a Markov chain of denoising steps); each step's variational lower bound is one of these ELBO terms. The original DDPM loss is Σ_t ELBO_t, derived by repeatedly applying Jensen.
The cleanest way to remember the ELBO: log p(x) - ELBO = D_KL(q(z) ‖ p(z|x)) ≥ 0. The ELBO is below log p(x) by exactly the KL divergence between your variational q and the true (intractable) posterior. Tightening q tightens the bound — this is the entire game of variational inference.
Tests · Verify cross-entropy of perfect model equals entropy. Verify KL divergence of perfect model is 0. Verify cross-entropy = entropy + KL divergence for all models.
#Live: Entropy, Cross-Entropy, and KL Side by Side
Run this cell to compute H(p), H(p, q), and KL(p‖q) for two discrete distributions, plot them side by side, then verify by hand that KL is NOT symmetric.
Loading visualization...
What Do You Think?
Suppose your model's predictions Q exactly match the true distribution P. What is KL(P‖Q)?
A Mathematical Theory of Communication
Claude Shannon (1948)
The paper that started it all. Surprisingly readable and accessible. Introduces entropy, channel capacity, and the fundamental limits of communication. Arguably the most important paper of the 20th century. Every ML practitioner should read at least the first few sections.
⚡ Playground:Probability Explorer → — change the probability distribution and watch entropy rise and fall.
Rare events carry more information. Information content is -log(P), so unlikely events are highly informative while certain events tell you nothing new
Entropy measures average surprise. A uniform distribution has maximum entropy (hardest to predict), while a distribution concentrated on one outcome has minimum entropy (easiest to predict)
Cross-entropy loss IS minimizing surprise. When you train a classifier with cross-entropy loss, you are forcing the model to be as unsurprised as possible by the true labels, which pushes predictions toward the true distribution
Confident wrong predictions are punished catastrophically. Since loss is -log(p) for the correct class, assigning near-zero probability to the truth produces enormous loss, driving the model toward well-calibrated confidence
KL divergence measures the gap between distributions. Cross-entropy equals entropy plus KL divergence, so minimizing cross-entropy is equivalent to minimizing the distance between your model's predictions and reality
A language model assigns probabilities: P('the')=0.4, P('a')=0.3, P('cat')=0.2, P('dog')=0.1. The true next word is 'cat'. What is the cross-entropy loss?
Congratulations -- you have completed the Mathematical Foundations track! You now have the building blocks for understanding everything in machine learning: vectors represent data, matrices transform it, eigenvalues reveal its structure, gradients guide optimization, probability quantifies uncertainty, gradient descent finds solutions, and information theory measures how well your model captures reality. Next stop: classical ML algorithms that put all of this math to work.