Skip to main content

RugvAI Labs

Tracks/Math Foundations/

Lesson 10 of 39

Your Reflection

Saves automatically

What’s one thing you learned? What’s still confusing?

0 characters

Was this lesson helpful?

How was the difficulty?

Lesson 10 of 3926%

Previous

Tensors, Broadcasting & Einsum

Up next

Vector & Matrix Calculus: Jacobian, Hessian & Backprop Rules

Continue Learning

Mathematical Foundations for ML

Vector & Matrix Calculus: Jacobian, Hessian & Backprop Rules

Jacobian, Hessian, key matrix-calculus identities, and a full end-to-end derivation of backprop for a softmax + cross-entropy classifier.

Mathematical Foundations for ML

Descriptive Statistics & Distributions

Mean, median, mode, standard deviation, and the distributions that appear everywhere in ML.

Mathematical Foundations for ML

Quantiles, Shape & Reading Real Data

Percentiles, quartiles, box plots, skewness, kurtosis, outliers and robust statistics, worked on one real-looking dataset.

Interactive Labs for This Track

Loss Landscape

Fly over the terrain your optimizer must navigate — peaks are bad, valleys are good

intermediate4 min

Vectors & Matrix Operations

Every neural network is just vectors being multiplied by matrices — build the intuition by dragging arrows on a coordinate plane.

Probability Distributions

Adjust μ, σ, n, p, and λ and watch the bell curve, bar chart, and shaded probability regions update live.

Discussion

Ask questions, share insights

Lesson 10 of 39

All lessons

1Math Toolkit 1: Functions, Exponents & Logs
2Math Toolkit 2: Sums, Symbols & Reading Formulas
3Vectors & Spaces
4Matrices as Transformations
5Eigenvalues & SVD
6Calculus: Derivatives & Gradients
7Optimization & Gradient Descent
8Linear Regression: Your First Model
9Tensors, Broadcasting & Einsum
10Computational Graphs & Autograd
11Vector & Matrix Calculus: Jacobian, Hessian & Backprop Rules
12Descriptive Statistics & Distributions
13Quantiles, Shape & Reading Real Data
14From Data to Distributions
15Counting & Probability Foundations
16Probability & Bayes' Theorem
17Random Variables, Expectation & Variance
18Checkpoint: Foundations Review
19Correlation: Pearson, Spearman & Multicollinearity
20Causation, Confounding & Simpson's Paradox
21The Distribution Zoo, Part 1: The Core Distributions
22The Distribution Zoo, Part 2: Fitting Data and Heavy Tails
23Sampling, Standard Error & the Central Limit Theorem
24The Bootstrap & Dependent Data
25Statistical Inference: Hypothesis Testing
26Power & Sample Size
27Multiple Testing, Peeking & P-Hacking
28Bayesian Inference in Practice
29Information Theory & Entropy
30Probability Inequalities & Monte Carlo
31Multivariate Gaussian & Joint Distributions
32MLE, MAP & Fisher Information
33Markov Chains & the Metropolis Sampler
34Markov Decision Processes
35Convex Optimization, KKT & Lagrangian Duality
36Noise, Scores & Diffusion Models
37Stochastic Calculus & SDEs (Advanced)
38Capstone Part 1: Describe, Test and Quantify StreamBox
39Capstone Part 2: Model It, Plan It, Decide

See AI think. Learn by watching machines learn — interactive ML visualizations from linear algebra to agents.

16 learning paths314 lessons

Learn

All Tracks
Start Here
Concept Index
Knowledge Graph

Explore

Playgrounds
Practice Problems
System Design
Search

Your Progress

Dashboard

Updates

Changelog

Support

Help Center

© 2026 RugvAI Labs. All rights reserved.

Interactive visualizations powered byD3.js·Three.js

Computational Graphs & Autograd

A computational graph is the record of every operation in a forward pass, and autograd is the chain rule applied to that record in reverse. Work a three-node example by hand, then build the 20-line engine behind loss.backward(). Afterwards, training a network is no longer magic: it is one forward walk and one backward walk over a DAG.

Where Part 1 left off. In Tensors, Broadcasting & Einsum you learned to read tensor shapes, to broadcast tensors of different shapes without loops, and to decode any einsum string. Those are the operations that sit inside a model. This lesson asks how a framework keeps track of them: it records each operation as a node in a graph, then walks that graph backward to produce gradients. No einsum is needed here, only the chain rule from Derivatives & Gradients.

Learning Objectives

After this lesson, you will be able to:

Explain a computational graph as a directed acyclic graph of operations, compute a gradient on one by hand, and describe why reverse-mode autodiff is the algorithm of choice for training neural networks
Describe what `loss.backward()` does in a framework, why reverse mode beats forward mode for training, and what memory it costs
Implement a minimal reverse-mode autograd engine (a `Value` class with one local backward function per operation) and verify its gradients against a hand calculation

Prerequisites

Before You Start

Why

Why This Matters

How loss.backward() does its magic in one line — PyTorch is secretly building a graph of every operation you do during the forward pass. When you call .backward(), it walks that graph in reverse and applies the chain rule node by node. This lesson makes that graph visible, and you will build a 20-line version yourself
Build this → Extend the Value class from the exercise with a relu and a pow operation, then check every gradient against a finite-difference estimate (f(x+h) - f(x)) / h.
Why deep models run out of memory. Reverse mode must remember every intermediate value from the forward pass. This lesson shows where that cost comes from and how gradient checkpointing pays it down.

#Computational Graphs

Before the formulas, three plain-words definitions. A graph here is just dots joined by arrows. The dots are called nodes and each node holds a number (or a tensor). Directed means every arrow has a direction, from an input to the result computed from it. Acyclic means that if you follow the arrows you can never return to where you started: no loops. A graph with directed arrows and no loops is a DAG (directed acyclic graph). Finally, a topological order is any order of visiting the nodes in which every node comes after all the nodes it depends on. Computing a formula from the inside out is exactly such an order.

Here is the smallest useful example, with real numbers. Take x = 2 and y = 3 and compute three things in turn: u = x * y, v = x + y, and L = u * v. The graph has the inputs x and y, then the nodes u, v and L. Going forward: u = 6, v = 5, L = 30. Note that x feeds two nodes (u and v) and so does y.

Now ask: how much does L change if x changes a little? Put the local slope (how much each node's output moves per unit change of one of its inputs) on every arrow:

L = u * v gives dL/du = v = 5 and dL/dv = u = 6.
u = x * y gives du/dx = y = 3. v = x + y gives dv/dx = 1.

There are two paths from x to L. Through u: 5 * 3 = 15. Through v: 6 * 1 = 6. Add the paths: dL/dx = 15 + 6 = 21. For y: through u, 5 * 2 = 10 (since du/dy = x = 2); through v, 6 * 1 = 6; total dL/dy = 16. Check with algebra: L = x²y + xy², so dL/dx = 2xy + y² = 12 + 9 = 21 and dL/dy = x² + 2xy = 4 + 12 = 16. The same numbers.

The picture below is this kind of graph. Numbers flow forward along the arrows, and the local slopes sit on the edges. Move an input and watch which node values change, then multiply the slopes along a path to see where each gradient comes from.

Loading visualization...

What we just did by hand is the general rule: the gradient of the final result with respect to an input equals the sum, over every path from that input to the result, of the product of the local slopes along the path.

\frac{\partial L}{\partial \theta_i} = \sum_{\text{paths } p \text{ from } \theta_i \text{ to } L} \prod_{e \in p} \frac{\partial \text{out}(e)}{\partial \text{in}(e)}

Intuition

Build Your Intuition

Real models are the same picture, only bigger. In Y = (A @ X + b).relu(); Z = Y @ W; loss = (Z - target).pow(2).mean(), the tensors A and X flow into a matmul node, which flows into an add node along with b, then relu, then another matmul with W, and so on until loss. The arrows never loop back, so it is a DAG.

Every modern deep-learning framework keeps this graph in memory as you compute. PyTorch builds it dynamically: each tensor remembers the operation that produced it (via tensor.grad_fn). JAX traces a function once and freezes the graph for compilation. Either way, the graph is the data structure that makes automatic differentiation possible.

The forward pass visits the nodes in topological order and computes each one from its parents. The result of the last node is your output (e.g., loss). The backward pass walks the same graph in reverse. Starting from loss with seed gradient dloss/dloss = 1, each node uses its own local slopes plus its already-computed children's gradients to compute the gradient for its parents. In the example: seed 1 at L, then u gets 5, v gets 6, then x collects 5*3 + 6*1 = 21.

#Forward mode vs reverse mode (a teaser)

Both modes implement the chain rule. The difference is direction and cost.

Forward mode computes derivatives in the same direction as the forward pass: it pushes a nudge (a "tangent vector", meaning one chosen direction in which to wiggle the inputs) forward through the graph. Cost: about one pass per input variable. Cheap if you have few inputs and many outputs.
Reverse mode (a.k.a. backpropagation) traverses the graph backward from a single output: it pulls a "cotangent vector" (the gradient seed, 1 for a scalar loss) back through the graph. Cost: about one pass total, regardless of how many inputs there are.

Neural networks have millions to billions of input parameters and one scalar output (the loss). That makes reverse mode a no-brainer: compute all gradients in roughly the cost of a single forward pass, instead of millions of forward passes.

What `loss.backward()` actually does

PyTorch's loss.backward() walks the computational graph in reverse, applying the chain rule node by node. This lesson's DAG is exactly the graph being walked. Every tensor with requires_grad=True has its .grad field populated with the corresponding partial derivative. The optimizer (SGD, Adam, etc.) then reads .grad and updates the parameter.

This is also why nn.Embedding(vocab, dim, sparse=True) exists: the gradient of the loss w.r.t. an embedding row is sparse. Only the rows for tokens that appeared in the batch get nonzero gradients (the others were never read during the forward pass, so the chain rule yields zero). A naive dense gradient tensor would be (vocab_size, dim), which for a large vocabulary is a lot of zeros to allocate. The sparse version stores only the rows that changed, cutting memory and compute for large embedding tables.

#Try It Yourself

One exercise: a 20-line autograd engine.

#Build autograd: 20 lines

Time to build the engine behind loss.backward(). Each Value holds a number, a grad slot, and the nodes it was made from. Each operation records a tiny function _back that pushes gradient to its parents using the local slopes from the worked example. Use += because a node like x can feed several others, and its gradient is the sum over paths. The test graph is the one you solved by hand: x = 2, y = 3, u = x * y, v = x + y, L = u * v.

pythonplayground.py · Pyodide

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

# A tiny autograd engine: 20 lines that do what loss.backward() does.
class Value:
  def __init__(self, data, parents=()):
      self.data = data          # the number this node holds
      self.grad = 0.0           # dL/d(this node), filled in by backward()
      self.parents = parents    # the nodes that were combined to make this one
      self._back = lambda: None # how to pass gradient to the parents

def __add__(self, other):
      out = Value(self.data + other.data, (self, other))
      def _back():
          # TODO 1: for z = a + b, the local slopes are 1 and 1.
          # Add (local slope * out.grad) into self.grad and other.grad (use +=).
          pass
      out._back = _back
      return out

def __mul__(self, other):
      out = Value(self.data * other.data, (self, other))
      def _back():
          # TODO 2: for z = a * b, dz/da = b and dz/db = a.
          # Add (local slope * out.grad) into self.grad and other.grad (use +=).
          pass
      out._back = _back
      return out

def backward(self):
      order, seen = [], set()
      def visit(node):          # depth-first: parents are listed before children
          if id(node) not in seen:
              seen.add(id(node))
              for p in node.parents:
                  visit(p)
              order.append(node)
      visit(self)
      self.grad = 1.0
      # TODO 3: walk `order` in REVERSE and call node._back() on each node.

x, y = Value(2.0), Value(3.0)
u = x * y
v = x + y
L = u * v
L.backward()
print("u, v, L =", u.data, v.data, L.data)
print("dL/dx =", x.grad, " dL/dy =", y.grad)
# TODO 4: check against the hand calculation: L = 30, dL/dx = 21, dL/dy = 16.

Tests · Verify u = 6, v = 5, L = 30, dL/dx = 21, dL/dy = 16, dL/du = 5, dL/dv = 6, and that the check against the hand calculation prints True.

With the TODOs filled in, the cell prints u, v, L = 6.0 5.0 30.0, then dL/dx = 21.0 dL/dy = 16.0, then dL/du = 5.0 dL/dv = 6.0, then matches hand calculation: True. Those are exactly the numbers from the path sums above. Before you fill in the TODOs the gradients print as 0.0, which is a good sign the skeleton is wired correctly. Notice what reversed(order) guarantees: a node's _back runs only after every node that depends on it has already pushed its gradient in.

Exit TicketIf you remember one thing

The gradient of the loss with respect to an input is the sum, over every path to the loss, of the product of local slopes along the path. Reverse mode computes all of those sums in one backward pass. That is the whole of loss.backward().

Take this with youNPTH-001

#Key Takeaways

Computational graphs make autodiff possible. Forward = topological evaluation of a DAG; backward = the gradient is the sum over paths of products of local slopes, computed in reverse. PyTorch builds it dynamically; JAX traces it once. loss.backward() is just one DAG traversal.
Reverse mode wins for training. A network has millions to billions of inputs and one scalar loss, so one backward pass gives every gradient. The price is memory, because the forward values must be kept or recomputed.
A tiny engine is enough. A Value class with one _back closure per operation, += accumulation and a reversed topological order reproduces loss.backward() on the worked example: dL/dx = 21 and dL/dy = 16.

#Quick Check

Quick Check1 / 4

Why is reverse-mode autodiff (backprop) the default for training neural networks instead of forward-mode?

Next up: Vector & Matrix Calculus: Jacobian, Hessian & Backprop. You have computed a gradient on a three-node graph by hand and built an engine that does it for you; next you will do the same with vectors and matrices, using Jacobians and the Hessian, and see backpropagation as one clean rule.