Neural Networks Dec 2015

Deep Residual Learning for Image Recognition

Kaiming He et al. · CVPR 2016

arXiv:1512.03385

In short

Very deep networks used to train worse than shallower ones. ResNet has each block learn only a change to its input, adding the input back through a shortcut, which let the authors train 152-layer networks that won ImageNet 2015.

Why it matters

Residual connections are now in nearly every deep network, including every transformer.

Read first

The 3 Field Guide ideas this paper leans on.

Neural Networks · Landmark Convolutional Neural Network A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects. Training · Standard Vanishing Gradient A problem where gradients become extremely small during backpropagation, preventing deep networks from learning effectively. Neural Networks · Standard Residual Connection Skip connections that allow gradients to flow directly through a network, enabling training of very deep networks (ResNet).

Starting from scratch? The full route 12 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Convolution ✓ understood
A mathematical operation that applies filters/kernels to input data to extract features like edges, textures, and patterns.
Convolutional Neural Network · read first ✓ understood
A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects.
Activation Function ✓ understood
A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Training ✓ understood
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
Loss Function ✓ understood
A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.
Gradient Descent ✓ understood
An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.
Backpropagation ✓ understood
The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent.
Vanishing Gradient · read first ✓ understood
A problem where gradients become extremely small during backpropagation, preventing deep networks from learning effectively.
Residual Connection · read first ✓ understood
Skip connections that allow gradients to flow directly through a network, enabling training of very deep networks (ResNet).

Nearby papers

Canon · 1997 Long Short-Term Memory LSTMs powered speech recognition, translation and text generation for twenty years, until transformers took over. Canon · 1998 Gradient-Based Learning Applied to Document Recognition It is the blueprint for the convolutional network, the architecture that later cracked computer vision. Canon · 2012 ImageNet Classification with Deep Convolutional Neural Networks This is the moment deep learning went mainstream: GPUs plus data plus depth beat decades of feature engineering. Canon · 2013 Playing Atari with Deep Reinforcement Learning It started deep reinforcement learning: one learner, raw perception, many tasks.