Neural Networks Dec 2012

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky et al. · NeurIPS 2012 (reprinted in Communications of the ACM, 2017)

Read the paper NeurIPS 2012 paper

doi:10.1145/3065386

In short

A large convolutional network, trained on two GPUs with ReLU activations and dropout, won the 2012 ImageNet challenge by a wide margin over every hand-engineered entry. The paper details the architecture and the tricks that made training at that scale work.

Why it matters

This is the moment deep learning went mainstream: GPUs plus data plus depth beat decades of feature engineering.

Read first

The 4 Field Guide ideas this paper leans on.

Neural Networks · Landmark Convolutional Neural Network A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects. Shipping AI · Standard GPU Graphics Processing Unit - hardware accelerator with thousands of cores, essential for parallel computation in deep learning. Neural Networks · Landmark Dropout A regularization technique that randomly deactivates neurons during training to prevent overfitting and improve generalization. Vision & Multimodal · Standard ImageNet A large-scale dataset of 14M images in 20K categories, historically used as the benchmark for image classification models.
Starting from scratch? The full route 15 ideas · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.

  3. Convolution ✓ understood

    A mathematical operation that applies filters/kernels to input data to extract features like edges, textures, and patterns.

  4. Convolutional Neural Network · read first ✓ understood

    A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects.

  5. GPU · read first ✓ understood

    Graphics Processing Unit - hardware accelerator with thousands of cores, essential for parallel computation in deep learning.

  6. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  7. Training Data ✓ understood

    The examples a model learns its weights from, kept separate from the validation and test data used to check how well it generalizes.

  8. Overfitting ✓ understood

    When a model fits its training data too closely, noise included, so it scores well on examples it has seen and poorly on new ones.

  9. Dropout · read first ✓ understood

    A regularization technique that randomly deactivates neurons during training to prevent overfitting and improve generalization.

  10. Supervised Learning ✓ understood

    Learning from examples paired with the correct answer, so a model can predict answers for new inputs it hasn't seen.

  11. Classification ✓ understood

    A supervised learning task where the model assigns each input to one of a fixed set of categories, such as spam or not spam.

  12. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  13. Computer Vision ✓ understood

    The field of AI that gets computers to extract meaning from images and video: what is in them, where it is, and how it moves.

  14. Image Classification ✓ understood

    Assigning a single label or category to an entire image, a fundamental computer vision task.

  15. ImageNet · read first ✓ understood

    A large-scale dataset of 14M images in 20K categories, historically used as the benchmark for image classification models.

Nearby papers

Canon · 1998 Gradient-Based Learning Applied to Document Recognition It is the blueprint for the convolutional network, the architecture that later cracked computer vision. Canon · 2009 ImageNet: A Large-Scale Hierarchical Image Database ImageNet and its annual challenge gave deep learning the benchmark it needed to prove itself. Canon · 2012 Improving Neural Networks by Preventing Co-adaptation of Feature Detectors Dropout became the standard regularizer of the deep-learning era and is still used inside large models. Canon · 2013 Playing Atari with Deep Reinforcement Learning It started deep reinforcement learning: one learner, raw perception, many tasks.
Summary in our own words; read the paper for the details. ← All papers