A derivative answers one question: "if I nudge the input a tiny bit, how much does the output move?" That's it. Every neural network learns by computing trillions of these tiny nudges and asking "did the answer get better?" If you've ever leaned on a steering wheel to correct your lane, you've already used calculus — this lesson just gives the move a name.
Learning Objectives
After this lesson, you will be able to:
Understand derivatives as the 'speedometer' of a function -- they tell you how fast something is changing right now
Compute partial derivatives (rate of change in one direction at a time) and combine them into a gradient that points uphill like a compass
See how the chain rule (connecting rates of change step by step) is the key idea behind how AI models learn from their mistakes
Apply the power rule, sum rule, and chain rule to compute gradients of composite functions like those found in neural network layers
You already understand derivatives -- you just did not know it. Every time you checked your speedometer, watched a stock price ticker, or noticed your phone battery draining faster than usual, you were thinking about rates of change. This lesson just gives that intuition a name and a formula.
This seemingly simple idea -- measuring how much the output wiggles when you wiggle the input -- is the mathematical engine behind all of machine learning. Every time an AI model learns from data, it computes derivatives to figure out which direction to adjust its settings.
Try it! Open the Python REPL (bottom-right of the screen: click Quick Actions, then Python) and type these lines yourself.
The derivative of a function f(x) at a point is the slope of the tangent line. Geometrically: "If I zoom in infinitely close, how steep is the curve here?"
f′(x)=h→0limhf(x+h)−f(x)
Below is a live tangent-line tracer. Drag the point along the curve and watch the slope (the derivative) update — that slope IS the derivative at that point.
The quotient rule unlocks one of the most important formulas in deep learning: the derivative of the sigmoid activation, σ(x) = 1 / (1 + e^(−x)). Every binary classifier and every gating mechanism in LSTMs, GRUs, and attention uses sigmoid; knowing where its derivative comes from removes the magic.
σ(x)=1+e−x1⇒σ′(x)=σ(x)(1−σ(x))
The result σ'(x) = σ(x)(1 − σ(x)) is what makes sigmoid practical: you compute the sigmoid output forward, then reuse it backward without re-evaluating the exponential. This kind of clean self-referential derivative shows up across DL — tanh'(x) = 1 − tanh²(x), the softmax gradient, and many gating updates have the same shape.
In ML, functions almost always depend on multiple variables -- a loss function depends on thousands or millions of parameters. A partial derivative measures the slope in one direction while holding everything else fixed.
The gradient bundles all partial derivatives into a single vector. It is the multi-dimensional generalization of the derivative:
∇f=∂x1∂f∂x2∂f⋮∂xn∂f
The key insight: the gradient points in the direction of steepest ascent. If you want to increase f as fast as possible, walk in the direction of the gradient. If you want to decrease f as fast as possible (which is what training a neural network does), walk in the opposite direction. That is gradient descent -- and it is the subject of our dedicated lesson.
In a transformer, ∇L w.r.t. an embedding row is sparse — only the rows for tokens that appeared in the batch get nonzero gradients. That's why nn.Embedding(vocab, dim, sparse=True) exists and why optimizer.zero_grad() must handle sparse tensors correctly.
What Do You Think?
At a local minimum of a function, what is the gradient?
At any critical point — a local minimum, a local maximum, or a saddle point — the ground is flat in every direction. Every partial derivative is zero, so the gradient is the zero vector. Zero gradient is a necessary but not sufficient condition for a minimum. This is how optimizers detect that they have stopped moving, but they still need second-order information (the Hessian) or empirical observation (loss not decreasing) to confirm it is the kind of stationary point you wanted. Saddle points — flat in some directions, downhill in others — are the dominant non-minimum trap in high dimensions and the focus of the gradient-descent lesson.
See how gradients guide optimization across a loss surface:
Try it: Watch gradients guide a ball downhill on a loss surfaceInteractive
Loading visualization...
Try this: Watch the gradient arrows on the surface. They always point uphill. The optimizer walks in the opposite direction (downhill). Notice how the arrows get shorter near the minimum -- the gradient shrinks as the surface flattens out.
Here is a top-down view of the gradient field — at every point on a 2D loss surface, an arrow shows you ∇f. Move your cursor around and notice how the arrows always point uphill, and how their length encodes steepness:
Loading visualization...
Quick check
For f(x, y) = 3x² + y², what is ∇f at the point (1, 2)?
Let f(x, y) = x^2 + 2xy. We want to find the gradient at any point (x, y). This is a simple function, but the process works identically for functions of millions of variables.
Treat y as a constant. The derivative of x^2 is 2x. The derivative of 2xy (treating y as a constant coefficient) is 2y. So the partial derivative with respect to x is 2x + 2y.
Now treat x as a constant. The derivative of x^2 with respect to y is 0 (no y involved). The derivative of 2xy with respect to y is 2x (treating x as the coefficient). So the partial derivative with respect to y is 2x.
The gradient is the vector of both partial derivatives: nabla f = [2x + 2y, 2x]. At the point (1, 3), the gradient is [2(1) + 2(3), 2(1)] = [8, 2]. This vector points in the direction of steepest increase of f at (1, 3).
The gradient [8, 2] at (1, 3) tells us: increasing x has 4 times the effect on f as increasing y at this point. If we wanted to decrease f, we would step in the direction [-8, -2] -- opposite the gradient. That is exactly one step of gradient descent.
#The Chain Rule: Derivatives Through Composed Functions
Most functions in ML are compositions -- the output of one function feeds into the next. The chain rule tells you how to compute derivatives through this chain.
Take y = sin(x²). This is a composition: an inner function u = x² feeds into an outer function y = sin(u). Apply the chain rule step by step:
Identify the layers. Outer: y = sin(u). Inner: u = x².
Differentiate each layer separately. dy/du = cos(u). du/dx = 2x.
Multiply. dy/dx = (dy/du)(du/dx) = cos(u) · 2x.
Substitute u back in. dy/dx = cos(x²) · 2x = 2x · cos(x²).
Sanity check at x = 0: dy/dx = 0 · cos(0) = 0. The function sin(x²) is flat at the origin -- correct, since sin(0) = 0 and the function only starts to lift off as x² grows. At x = √(π/2) ≈ 1.253, x² = π/2, so cos(x²) = 0 and dy/dx = 0 again -- correct, that is where sin(x²) hits its first peak.
This same three-step pattern -- peel off the outermost layer, multiply by the derivative of what is left -- is exactly how backprop walks through a 100-layer transformer. The math does not get harder with more layers; you just have a longer chain of multiplications.
Why this matters for ML: a neural network is a giant chain of composed functions. Layer 1 feeds into Layer 2 feeds into Layer 3 feeds into the loss function. Backpropagation is literally the chain rule applied backwards through this composition, computing how the loss changes with respect to each parameter in every layer.
Chain Rule & Backprop — Run Forward Pass, Then Watch Gradients Flow BackInteractive
Loading visualization...
∂w1∂L=∂a3∂L⋅∂a2∂a3⋅∂a1∂a2⋅∂w1∂a1
#Taylor Series: Approximating Any Function With Derivatives
Once you have derivatives, you can approximate any smooth function by an infinite sum of polynomial terms. The Taylor series says: if you know the value, slope, and curvature of a function at a point, you can approximate the whole function nearby.
In the playground below, drag the expansion point and the number of terms. Watch the polynomial cling tightly to the function near the expansion point, then peel off as you move away:
Loading visualization...
This matters in ML for two reasons. First, the second-order Taylor expansion is the basis of all "Newton-style" optimizers (the Hessian is just the next term after the gradient). Second, when researchers analyze how a neural net behaves around a trained minimum, they almost always look at the first few Taylor terms — the gradient (first-order) and the Hessian (second-order) — to understand local geometry.
Quick check
Apply the chain rule: if y = (3x + 2)⁵, what is dy/dx?
Tests · Verify numerical derivatives match exact values within 1e-5. Try modifying f(x) to x^2 and confirm f'(x) = 2x.
Learning representations by back-propagating errors
David Rumelhart, Geoffrey Hinton, Ronald Williams (1986)
The paper that popularized backpropagation for training neural networks. Showed that the chain rule, applied systematically through a network, could learn useful internal representations. This single idea enabled the deep learning revolution.
Let's stop computing derivatives by hand and let SymPy do it. The playground below uses real symbolic differentiation — change the function and watch the derivative recompute. This is the same engine that powers backprop in WolframAlpha and Mathematica.
Derivatives measure sensitivity. A derivative tells you how much the output changes when you nudge the input by a tiny amount, acting as a "speedometer" for any function
The gradient bundles all partial derivatives. For multi-variable functions, the gradient is a vector pointing in the direction of steepest ascent, and its magnitude tells you how steep that ascent is
The chain rule enables backpropagation. Because neural networks are compositions of functions (layers), the chain rule lets you compute how the loss changes with respect to any parameter, no matter how deep in the network
Automatic differentiation handles the complexity. Frameworks like PyTorch track computations on a graph and apply the chain rule in reverse automatically, so you never compute gradients by hand in practice
At a minimum, the gradient is zero. Optimizers detect convergence when gradients become very small, indicating the loss surface is flat (though this could also indicate a saddle point)
What does the gradient of a function point toward?
Next up: Probability and Bayes' Theorem -- learning to reason under uncertainty. You will discover why a 99% accurate test can still be wrong most of the time, and how Bayesian thinking is the foundation of classification.