Rebuilding AlphaGo: A Deep Dive into AI Go Core Principles and Implications for LLMs

📝 Podcast Introduction

This episode is a clone of the deep interview from the popular tech podcast *Dwarkesh Patel Podcast* titled "What rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs - Eric Jang". The host, Dwarkesh Patel, engages in a fascinating technical dive with guest Eric Jang, reconstructing the core ideas of AlphaGo from scratch using modern open-source tools and minimal budget.

Eric Jang previously served as the VP of AI at 1X Technologies and was an advanced research scientist at Google DeepMind Robotics. During his leave, he independently reconstructed and improved AlphaGo, writing a detailed technical tutorial that has garnered significant community attention. He is renowned for his unique insights into AlphaGo's core mechanisms and his pioneering thoughts on automated AI research.

⏱️ Timestamps

00:00 Opening & Podcast Introduction Understanding AlphaGo from Scratch

02:05 Why AlphaGo is So Fascinating: A Neural Network That "Solves" Almost Impossible Search

03:43 Go Rules Quick Summary: From Taking Pieces to Trump-Taylor Scoring

08:38 Search Tree and Explosion: 361^300, Larger Than the Number of Atoms in the Universe

Monte Carlo Tree Search (MCTS) Core Principles

11:16 UCB and PUCT: How to Explore While Building the Tree

15:59 Value Function Arrives: The Intuition of Humans "Seeing the Win/Loss" in One Look, AI Can Also Have It

21:02 Strategy Network: Guessing Where to Search, Dramatically Pruning

Neural Networks and Search: A Perfect Match

24:54 Four Steps of MCTS: Selection, Expansion, Evaluation, Backpropagation

27:28 Architecture Choice: Why ResNet Still Outperforms Transformer in Low Budget

34:23 The Magic of Initialization: Teaching the Model What a Good Game Is with Human Chess Patterns

42:21 Self-Play Loop: Letting Search Feed the Network for Strategy Iteration

The Elegance and Cruelty of Reinforcement Learning

47:41 MCTS as an Improvement Operator: Always Giving a Better Answer Than the Current Policy

52:00 Knowledge Distillation: Internalizing Thousands of Search Steps into a Single Forward Pass

57:04 Training Techniques for Value Function: Small Board Pretraining and Final Game Labels

01:03:01 Deep Impact: How 10 Layers of Neural Networks Solve NP Hard Problems

01:11:35 Comparing LLM RL: Why Variance Explodes, and the "Pipe Absorbing Signals" Dilemma

01:22:21 Can MCTS Be Used Directly for LLM Reasoning? Challenges of Breadth, Depth, and Action Space

Computational Efficiency and Automated Research

01:28:41 Scaling Calculations: From Millions to Thousands of Dollars, AlphaGo Became Cheaper

01:38:08 Off-Policy Training and Replay Buffer: How to Reuse Old Data

01:47:04 Information Theory Perspective: Supervised Learning Has Much More Information per Sample than RL

01:55:36 Go as an AI Scientist's Training Ground: Using Outer Cycles to Validate Research Intuitions

02:05:12 Research Taste and Verifiability: How to Design a Correct RL Environment

02:08:03 Closing & Resource Recommendations

🌟 Highlights

💡 10-Layer Network, Solving NP Hard Problems Eric points out that AlphaGo's most profound contribution is not the game itself, but a conceptual breakthrough: just 10 layers of neural networks can, through a single forward pass, approximate a deeply complex search problem with high accuracy. This suggests that macroscopic features can dissolve our traditional understanding of computational complexity, similar to what we see in models like AlphaFold.

“这是一个突破，我觉得今天大多数人都没能完全领会它有多么深远。”

🛠️ MCTS's Elegance: Never Starting from 0% Unlike the naive gradient-based methods used by today's LLMs, AlphaGo's MCTS always provides an improved policy label based on the current state. This means its learning process never gets stuck in a "zero signal" desert, with clear supervisory goals at every step, leading to extraordinary sampling efficiency and stability.

“AlphaGo 之所以优雅，就是你永远不需要从一个 0% 的成功率开始，也不需要解决怎么拿到非零成功率的探索问题。”

🚀 Supervised Learning Information Efficiency Wins Eric and Dwarkesh compare supervised learning and RL from an information theory perspective. In regions with low pass rates, RL can only provide very little learning bits per sample, while supervised learning, through soft labels (the entire probability distribution), can provide much more information. This explains why distillation is so effective—MCTS's access count distribution serves as a soft target, passing on far more "hidden knowledge" than a single action label.

“在一个软标签里，每样本的信息量，以比特计，要大得多。这就是为什么蒸馏这么有效。”

⚖️ Become First, Calculating Power is Always Most Expensive Eric shares his experience of reconstructing AlphaGo using just $10,000 of computing power, contrasting it with DeepMind's usual millions of dollars and custom TPU clusters. He emphasizes, “The computing power needed to become the first to do something is always much greater than the power required to catch up later.” This principle also holds true in the age of LLMs, where pioneers must pay a significant premium for exploring unknown territory.

🧪 Go as an AI Scientist's Training Ground Eric is currently building a Go environment as an "outer loop" to train automated AI research agents. Because Go validates quickly and has clear win/loss outcomes, it offers a low-cost way to test the ability of agents to propose hypotheses, design experiments, and explain results, potentially leading to applications in more complex scientific discovery tasks.

“我搭建这个围棋环境的动机之一，就是觉得围棋承载了大量非常有趣的研究问题，而且验证速度很快。”

🌐 Podcast Information Supplement Translated from the *Dwarkesh Patel Podcast*

This episode uses AI voice cloning technology to translate the voices of the original host and guest into Chinese, which may result in slight differences in pronunciation.

AI translation was used, so there might be some awkwardness in certain parts;

If you'd like to listen to other foreign podcasts in Chinese in the future, please contact us via WeChat: iEvenight.