Neural Networks Nov 1998

Gradient-Based Learning Applied to Document Recognition

Yann LeCun et al. · Proceedings of the IEEE

doi:10.1109/5.726791

In short

LeCun and colleagues show that a convolutional network trained end to end with backpropagation reads handwritten digits better than hand-engineered pipelines. The paper also presents LeNet-5 and a system that read a large share of the cheques written in the US.

Why it matters

It is the blueprint for the convolutional network, the architecture that later cracked computer vision.

Read first

The 4 Field Guide ideas this paper leans on.

Neural Networks · Landmark Neural Network A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections. Training · Landmark Backpropagation The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent. Neural Networks · Standard Convolution A mathematical operation that applies filters/kernels to input data to extract features like edges, textures, and patterns. Neural Networks · Landmark Convolutional Neural Network A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects.

Starting from scratch? The full route 9 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network · read first ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Training ✓ understood
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
Loss Function ✓ understood
A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.
Gradient Descent ✓ understood
An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.
Backpropagation · read first ✓ understood
The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent.
Convolution · read first ✓ understood
A mathematical operation that applies filters/kernels to input data to extract features like edges, textures, and patterns.
Convolutional Neural Network · read first ✓ understood
A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects.

Nearby papers

Canon · 1958 The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain It is the first learning neural network: the idea that weights, not rules, should hold the knowledge starts here. Canon · 1986 Learning Representations by Back-Propagating Errors Backpropagation is how essentially every neural network, from LeNet to today’s LLMs, is trained. Canon · 2012 ImageNet Classification with Deep Convolutional Neural Networks This is the moment deep learning went mainstream: GPUs plus data plus depth beat decades of feature engineering. Canon · 2013 Playing Atari with Deep Reinforcement Learning It started deep reinforcement learning: one learner, raw perception, many tasks.