Standard 12 stops to get here

PPO

Proximal Policy Optimization - a stable and efficient policy gradient algorithm widely used in RLHF for training LLMs.

Your route here

12 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Reinforcement Learning ✓ understood

    Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.

  3. Agent ✓ understood

    In RL, the learner or decision-maker that takes actions in an environment to maximize cumulative reward.

  4. Policy ✓ understood

    A strategy or mapping from states to actions that defines the agent's behavior in reinforcement learning.

  5. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  6. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  7. Loss Function ✓ understood

    A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.

  8. Gradient Descent ✓ understood

    An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

  9. Policy Gradient ✓ understood

    RL methods that directly optimize the policy by computing gradients of expected reward with respect to policy parameters.

  10. Reward ✓ understood

    A scalar feedback signal indicating how good an action was, used to train reinforcement learning agents.

  11. Value Function ✓ understood

    A function estimating expected cumulative reward from a state (state-value) or state-action pair (action-value/Q-value).

  12. Actor-Critic ✓ understood

    RL architecture with two components: an actor (policy) that selects actions and a critic (value function) that evaluates them.

  13. PPO · you are here ✓ understood

Where it sits

Before this

Policy Gradient Actor-Critic
PPO

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Language & LLMs RLHF Reinforcement Learning from Human Feedback - training models using human preferences to align behavior with human values. Language & LLMs Constitutional AI Training AI systems using principles and rules rather than only human feedback, developed by Anthropic for Claude. Agents & RL Q-Learning A model-free RL algorithm that learns action-value functions (Q-values) to determine optimal actions in each state. Agents & RL Deep Q-Network Combining Q-learning with deep neural networks to handle high-dimensional state spaces, enabling RL for complex tasks like Atari games. Agents & RL Reward A scalar feedback signal indicating how good an action was, used to train reinforcement learning agents.

In the research

All papers →

A paper that builds on PPO .