Landmark 6 stops to get here · leads to 5

Backpropagation

The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent.

Your route here

6 stops · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Training ✓ understood
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
Loss Function ✓ understood
A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.
Gradient Descent ✓ understood
An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.
Backpropagation · you are here ✓ understood

Picture it

01 Forward pass Inputs flow through to a prediction
02 Compute loss Compare prediction to the target
03 Gradient at output How the loss changes with the output
04 Propagate backward Chain rule, layer by layer
05 Gradient per weight Handed to gradient descent

Notice how the error signal travels backward, reusing each layer's gradient to compute the one before it.

Where it sits

Before this

Neural Network Gradient Descent

Backpropagation

Leads to

Saliency Map Vanishing Gradient Exploding Gradient Gradient Accumulation Gradient Checkpointing

Explore nearby

Neural Networks Backward Pass The process of computing gradients by propagating error signals backward through the network during training. Neural Networks Forward Pass The process of passing input through the network to generate predictions during training or inference. Training Vanishing Gradient A problem where gradients become extremely small during backpropagation, preventing deep networks from learning effectively. Training Exploding Gradient A problem where gradients become extremely large during backpropagation, causing unstable training and NaN values. Training Gradient Checkpointing Trading computation for memory by recomputing activations during backprop instead of storing them.

In the research

2 papers that build on Backpropagation .

Canon · 1986 Learning Representations by Back-Propagating Errors Backpropagation is how essentially every neural network, from LeNet to today’s LLMs, is trained.

Canon · 1998 Gradient-Based Learning Applied to Document Recognition It is the blueprint for the convolutional network, the architecture that later cracked computer vision.