AlphaZero: Mastering Games with Self-Play Reinforcement Learning

AlphaZero, a general reinforcement learning algorithm, achieved superhuman performance in chess, shogi, and Go. It learned solely through self-play, starting from random moves and given only the game rules.

This groundbreaking AI defeated world-champion programs like Stockfish (chess) and Elmo (shogi) within 24 hours of training. It represents a significant leap from previous game-playing AIs.

Traditional chess programs rely on sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions refined by human experts over decades. AlphaZero takes a different path.

Unlike Deep Blue, which defeated Kasparov in 1997 using vast search trees and human-tuned heuristics, AlphaZero replaces this handcrafted knowledge with deep neural networks and a tabula rasa approach.

AlphaZero's core is a deep neural network that takes a board position as input and outputs move probabilities and an estimated game outcome. This network is trained entirely from self-play games.

Instead of alpha-beta search, AlphaZero employs a general-purpose Monte-Carlo Tree Search (MCTS). Each search simulates games of self-play, guided by the neural network's predictions.

The neural network parameters are continuously updated via gradient descent, minimizing the error between predicted outcomes and actual game results, and maximizing similarity to search probabilities.

Key differences from AlphaGo Zero include: AlphaZero optimizes for expected outcomes (handling draws), does not exploit game symmetries (chess/shogi are asymmetric), and updates its network continuously.

AlphaZero uses the same algorithm settings, network architecture, and hyperparameters for all three games. This demonstrates its generality and adaptability across diverse and complex domains.

Despite searching far fewer positions per second (80k for chess vs. Stockfish's 70M), AlphaZero compensates by focusing selectively on promising variations, a more 'human-like' search approach.

AlphaZero's MCTS also scaled more effectively with thinking time than traditional alpha-beta engines, challenging the long-held belief that alpha-beta search is inherently superior in these domains.

The AI independently discovered and frequently played common human chess openings during its self-play training, showcasing its ability to master a wide spectrum of strategic play from first principles.