Tensors & Automatic Differentiation

Don't worry if "tensor" and "autograd" sound intimidating — by the end of this lesson you will be able to explain both to a friend in plain English, and the rest of deep learning will feel like just a careful application of these two ideas.

#What Is a Tensor, Really?

A tensor has four properties you constantly check:

• shape -- the tuple of dimensions (e.g., (32, 3, 224, 224) for a batch of 32 RGB images at 224x224)
• dtype -- the numeric type (float32 is the default for training, float16 or bfloat16 for mixed precision, int64 for indices)
• device -- where the data lives (cpu, cuda:0, mps)
• requires_grad -- whether autograd should track operations on it

#Shapes and Broadcasting

When two tensors have different shapes, broadcasting is the rule for whether the operation is legal and how it expands. The rule, read right-to-left:

1. Each dimension matches if the sizes are equal.
2. A dimension also matches if one side is 1 (it is "stretched" to the other side's size).
3. Missing dimensions on the left are treated as 1.

#Reshape vs. View vs. Contiguous

Three operations that look similar but differ in whether they touch memory:

• .view() -- returns a new tensor that shares the same memory, just reinterpreted with a new shape. Cheap. Only works if the tensor is contiguous.
• .reshape() -- prefers to return a view, but copies if it must.
• .contiguous() -- forces a copy into row-major (C-order) memory. You need this after .transpose() before calling .view().

Try it! Open the Python REPL and type these lines yourself. import torch; x = torch.arange(12); print(x.view(3,4)) -- you just reshaped a flat tensor into a 3×4 grid without copying a single byte.

#Why Do We Need Automatic Differentiation?

To train a neural network, you need gradients of a loss with respect to every parameter. Three options exist:

1. By hand -- write the derivative on paper, code it up. Doable for tiny models, suicidal for anything real. One bug in the chain rule and your model silently learns garbage.
2. Numerical (finite differences) -- estimate df/dx as (f(x+h) - f(x-h)) / (2h). Easy to write but catastrophically slow (you need one forward pass per parameter), and numerically unstable.
3. Automatic differentiation -- the framework records every operation in a graph and applies the chain rule mechanically. Exact gradients, one backward pass for all parameters, no hand-derived math.

Every modern deep-learning framework picks option 3.

#Forward Mode vs. Reverse Mode

There are two ways to apply the chain rule. The difference matters for one reason: efficiency.

For deep learning — which always has many parameters and one scalar loss — reverse mode is overwhelmingly the right choice. When practitioners say "autograd," they mean reverse-mode autograd.

#Dynamic vs. Static Graphs

Frameworks split on a design choice:

• Static graph (TensorFlow 1.x, ONNX): you define the entire computation graph up front, then feed data into it. Optimizable, harder to debug.
• Dynamic graph (PyTorch, TF 2.x eager, JAX with jit-tracing): the graph is built as Python runs. Pythonic loops and conditionals just work; printf-debugging works. Slightly less optimization headroom — but in practice, JIT compilers (torch.compile, jax.jit) close most of the gap.

PyTorch won the deep learning research community largely because its dynamic graph made debugging and prototyping fast. Static-graph frameworks survive in production deployment scenarios where every microsecond matters.

#The Four Autograd Patterns

PyTorch's autograd API boils down to four moves you make in every training script:

1. requires_grad=True -- Turn On Tracking

Mark a tensor as needing gradients. From this point, every operation involving it is logged.

w = torch.randn(3, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
x = torch.tensor([1.0, 2.0, 3.0])  # input — no grad needed

y = (w * x).sum() + b  # autograd silently records every op

2. .backward() -- Run the Chain Rule

y.backward()
print(w.grad)  # tensor([1., 2., 3.])  — equals x
print(b.grad)  # tensor([1.])

3. .grad -- Read the Result

4. .zero_grad() -- Reset Between Steps

optimizer.zero_grad()  # essential — clears every parameter's .grad
loss.backward()
optimizer.step()

The correct answer is the second one. PyTorch's API was designed for accumulation (it makes implementing RNNs and gradient accumulation across micro-batches trivial), but it punishes forgetfulness. After 100 batches without zeroing, every gradient is roughly 100x what it should be — your effective learning rate is wrong by 2 orders of magnitude, and within a few steps the weights diverge to NaN.

#Building Autograd From Scratch (Karpathy's Micrograd)

If you understood that 50-line file, you understand the soul of every deep learning framework ever built. PyTorch differs in three ways: it operates on tensors instead of scalars, it runs on the GPU, and it has thousands of operation types — but the core algorithm is exactly what you just wrote.

#The Detach / no_grad / .data Triangle

Three operations that all "stop tracking gradients" but mean different things:

• x.detach() -- returns a new tensor sharing the same data but cut from the autograd graph. The recommended way.
• with torch.no_grad(): -- a context manager that disables autograd globally for everything inside. Use during inference and validation.
• x.data -- direct access to the underlying storage, bypassing autograd. Almost always the wrong choice — mutations through .data are invisible to autograd and can silently corrupt training.

#Devices: CPU, CUDA, MPS

Tensors live on a device. Moving them is a one-liner; forgetting to move them is a one-line crash.

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
model = model.to(device)
x = x.to(device)
y = y.to(device)
# Every input AND every parameter must be on the same device

The two error messages you will see most often:

• RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! -- you forgot .to(device) somewhere.
• RuntimeError: CUDA out of memory. -- your batch size is too large for your GPU. Lower it, or use gradient accumulation, or switch to mixed precision (covered later in this track).

#Key Takeaways

• A tensor is NumPy plus GPU plus gradient tracking -- four properties (shape, dtype, device, requires_grad) determine its behavior, and most beginner bugs come from a mismatch in one of them
• Reverse-mode autograd is "every op logs itself, then chain-rule walks the log backwards" -- one forward + one backward pass gives you gradients for all parameters, which is why every modern DL framework uses it
• The four autograd patterns are requires_grad, .backward(), .grad, .zero_grad() -- master these four moves and you can train any function-shaped thing without writing a derivative by hand
• Detach when you want gradients to stop, no_grad for inference, never .data -- .data bypasses autograd silently and produces correct-looking but wrong gradients downstream
• Karpathy's micrograd shows you the whole idea in 100 lines -- if you internalize it, the rest of deep learning becomes a careful, GPU-accelerated, parallel version of one elegant algorithm

#Quick Check