Standard 11 stops to get here

AdamW

Adam with decoupled weight decay, providing better regularization and often superior performance.

Your route here

11 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training Data ✓ understood

    The examples a model learns its weights from, kept separate from the validation and test data used to check how well it generalizes.

  3. Overfitting ✓ understood

    When a model fits its training data too closely, noise included, so it scores well on examples it has seen and poorly on new ones.

  4. Regularization ✓ understood

    Techniques to prevent overfitting by adding constraints or penalties to the model (L1, L2, dropout, early stopping).

  5. Weight Decay ✓ understood

    A regularization technique that shrinks weights toward zero during optimization. Equivalent to L2 regularization in standard SGD, but differs when using adaptive optimizers like Adam.

  6. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  7. Loss Function ✓ understood

    A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.

  8. Gradient Descent ✓ understood

    An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

  9. Momentum ✓ understood

    An optimization technique that accelerates gradient descent by accumulating past gradients, helping escape local minima.

  10. Learning Rate ✓ understood

    A hyperparameter controlling the step size in gradient descent - too high causes instability, too low slows convergence.

  11. Adam Optimizer ✓ understood

    An adaptive learning rate optimization algorithm combining momentum and RMSprop, widely used for training neural networks.

  12. AdamW · you are here ✓ understood

Adam with decoupled weight decay, providing better regularization and often superior performance.

This concept is essential for understanding training & optimization and forms a key part of modern AI systems.

  • Adam Optimizer
  • Weight Decay
  • Optimizer

Where it sits

Before this

Adam Optimizer Weight Decay
AdamW

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Training L2 Regularization Adding the sum of squared weights to the loss function, penalizing large weights and improving generalization. Training LAMB Optimizer Layer-wise Adaptive Moments optimizer for Batch training - enables very large batch training for transformers. Training Lookahead Optimizer A wrapper that maintains fast and slow weights, periodically updating slow weights, improving convergence. Training Stochastic Gradient Descent A variant of gradient descent that updates parameters using gradients computed on a single random training example at a time (though often used to refer to mini-batch gradient descent).