Agents & RL Jan 2016

Mastering the Game of Go with Deep Neural Networks and Tree Search

David Silver et al. · Nature

doi:10.1038/nature16961

In short

AlphaGo combines a policy network that suggests moves and a value network that judges positions with Monte Carlo tree search. Trained on expert games and then by playing itself, it became the first program to beat a professional Go player on a full board.

Why it matters

It proved that learned intuition plus search can master a problem long thought a decade away.

Read first

The 4 Field Guide ideas this paper leans on.

Agents & RL · Landmark Reinforcement Learning Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies. Agents & RL · Landmark Policy A strategy or mapping from states to actions that defines the agent's behavior in reinforcement learning. Agents & RL · Standard Value Function A function estimating expected cumulative reward from a state (state-value) or state-action pair (action-value/Q-value). Agents & RL · Reference Monte Carlo Tree Search A search algorithm combining tree search with random sampling, used in game-playing AIs.

Starting from scratch? The full route 8 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Reinforcement Learning · read first ✓ understood
Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.
Agent ✓ understood
In RL, the learner or decision-maker that takes actions in an environment to maximize cumulative reward.
Policy · read first ✓ understood
A strategy or mapping from states to actions that defines the agent's behavior in reinforcement learning.
Reward ✓ understood
A scalar feedback signal indicating how good an action was, used to train reinforcement learning agents.
Value Function · read first ✓ understood
A function estimating expected cumulative reward from a state (state-value) or state-action pair (action-value/Q-value).
Exploration vs Exploitation ✓ understood
The RL dilemma of trying new actions (exploration) versus using known good actions (exploitation) to maximize reward.
Monte Carlo Tree Search · read first ✓ understood
A search algorithm combining tree search with random sampling, used in game-playing AIs.

Nearby papers

Canon · 2013 Playing Atari with Deep Reinforcement Learning It started deep reinforcement learning: one learner, raw perception, many tasks. Canon · 2017 Proximal Policy Optimization Algorithms PPO became the default RL algorithm, including for RLHF on the first ChatGPT-era models. Canon · 2022 Training Language Models to Follow Instructions with Human Feedback This recipe, RLHF, is what turned raw language models into assistants like ChatGPT. Frontier · #3 InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency It narrows the gap between open and commercial multimodal models on reasoning and agent tasks.