--- title: "How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope" source_name: "Dwarkesh Patel" original_url: "https://www.youtube.com/watch?v=xmkSf5IS-zw" canonical_url: "https://www.traeai.com/articles/3ea50b96-aa80-47eb-ad8f-25c4bed3a88b" content_type: "video" language: "其他" score: 0 tags: [] published_at: "2026-04-29T17:20:27+00:00" created_at: "2026-04-30T09:35:42.030571+00:00" --- # How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope Canonical URL: https://www.traeai.com/articles/3ea50b96-aa80-47eb-ad8f-25c4bed3a88b Original source: https://www.youtube.com/watch?v=xmkSf5IS-zw ## Summary 该链接指向的是一个YouTube视频页面，而非一篇技术文章。无法从中提取有效信息。 ## Key Takeaways - 该链接指向的是一个YouTube视频页面，而非一篇技术文章 - 无法从中提取有效信息 ## Content Title: How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope URL Source: http://www.youtube.com/watch?v=xmkSf5IS-zw Markdown Content: # How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope - YouTube Back [![Image 1](http://www.youtube.com/watch?v=xmkSf5IS-zw)](http://www.youtube.com/ "YouTube Home") Skip navigation Search Search with your voice [](http://www.youtube.com/watch?v=xmkSf5IS-zw) [Sign in](https://accounts.google.com/ServiceLogin?service=youtube&uilel=3&passive=true&continue=https%3A%2F%2Fwww.youtube.com%2Fsignin%3Faction_handle_signin%3Dtrue%26app%3Ddesktop%26hl%3Den%26next%3Dhttps%253A%252F%252Fwww.youtube.com%252Fwatch%253Fv%253DxmkSf5IS-zw&hl=en&ec=65620) [![Image 2](http://www.youtube.com/watch?v=xmkSf5IS-zw)](http://www.youtube.com/ "YouTube Home") [](http://www.youtube.com/watch?v=xmkSf5IS-zw) [](http://www.youtube.com/watch?v=xmkSf5IS-zw) [](http://www.youtube.com/watch?v=xmkSf5IS-zw) [](http://www.youtube.com/watch?v=xmkSf5IS-zw) [How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope](http://www.youtube.com/watch?v=xmkSf5IS-zw) [](http://www.youtube.com/watch?v=xmkSf5IS-zw) Tap to unmute 2x [![Image 3](http://www.youtube.com/watch?v=xmkSf5IS-zw)](http://www.youtube.com/watch?v=xmkSf5IS-zw) ## How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope Dwarkesh Patel 80,828 views 16 hours ago [](http://www.youtube.com/watch?v=xmkSf5IS-zw) Search Copy link Info Shopping ![Image 4](http://www.youtube.com/watch?v=xmkSf5IS-zw) [![Image 5](http://www.youtube.com/watch?v=xmkSf5IS-zw)](http://www.youtube.com/watch?v=xmkSf5IS-zw) If playback doesn't begin shortly, try restarting your device. • You're signed out Videos you watch may be added to the TV's watch history and influence TV recommendations. To avoid this, cancel and sign in to YouTube on your computer. Cancel Confirm [](http://www.youtube.com/watch?v=xmkSf5IS-zw) Share [](http://www.youtube.com/watch?v=xmkSf5IS-zw "Share link")- [x] Include playlist An error occurred while retrieving sharing information. Please try again later. ![Image 6](http://www.youtube.com/watch?v=xmkSf5IS-zw) 0:00 [](http://www.youtube.com/watch?v=xmkSf5IS-zw)[](https://www.youtube.com/watch?v=6eBSHbLKuN0 "Next (SHIFT+n)") 0:00 / 0:00 Live •Watch full video • How batch size affects token cost and speed • [29:49 Andrej Karpathy: From Vibe Coding to Agentic Engineering Sequoia Capital 151K views • 18 hours ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=96jN2OCOfLs)[40:57 Demis Hassabis: Agents, AGI & The Next Big Scientific Breakthrough Y Combinator 50K views • 19 hours ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=JNyuX1zoOgU)[28:07 Mastering Claude Code in 30 minutes Anthropic 1.1M views • Streamed 11 months ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=6eBSHbLKuN0)[1:43:13 Jensen Huang – Will Nvidia’s moat persist?Dwarkesh Patel 733K views • 2 weeks ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=Hrbq66XqtCo)[29:43 AI Chatbots: Last Week Tonight with John Oliver (HBO)LastWeekTonight 2.8M views • 3 days ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=Ykvf3MunGf8)[2:49:46 Elon Musk – "In 36 months, the cheapest place to put AI will be space”Dwarkesh Patel and Stripe 1.3M views • 2 months ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=BYXbuik3dgA)[58:20 Sam Altman in conversation with Patrick Collison Stripe 6.3K views • 6 hours ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=1ySBxK7viNs)[33:25 The disaster I never imagined having to worry about Veritasium 1.2M views • 13 hours ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=ksn5yrsC3Wg)[26:52 Demis Hassabis: We're Three Quarters of the Way to AGI Sequoia Capital 47K views • 18 hours ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=AFpeWo1GTeg)[1:33:56 Building Pi, and what makes self-modifying software so fascinating The Pragmatic Engineer 16K views • 19 hours ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=n5f51gtuGHE)[24:02 Humanoid Robots and the Gap Between Hype and Reality | Bloomberg Primer Bloomberg Originals 101K views • 1 day ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=UQZooauU-FQ)[2:22:20 Dario Amodei — “We are near the end of the exponential”Dwarkesh Patel 1M views • 2 months ago Live Playlist ()Mix (50+)](https://www.youtube.com/watch?v=n1E9IZfvGMA) Sign in to confirm you’re not a bot This helps protect our community. [Learn more](https://support.google.com/youtube/answer/3037019#zippy=%2Ccheck-that-youre-signed-into-youtube) [Sign in](https://accounts.google.com/ServiceLogin?service=youtube&uilel=3&passive=true&continue=https%3A%2F%2Fwww.youtube.com%2Fsignin%3Faction_handle_signin%3Dtrue%26app%3Ddesktop%26hl%3Den%26next%3D%252Fwatch%253Fv%253DxmkSf5IS-zw&hl=en) # How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope [![Image 7](https://yt3.ggpht.com/lG-z7sTfhFIW2Ne1oXMHvXMXyZSaA02_I17gUel0GAEj7OypsSHQ7PE91Vp4bTbpm3PTIAWJdko=s48-c-k-c0x00ffffff-no-rj)](http://www.youtube.com/@DwarkeshPatel) [Dwarkesh Patel](http://www.youtube.com/@DwarkeshPatel) Dwarkesh Patel 1.29M subscribers Subscribe Subscribed 3.4K Share Save Download Download 80,828 views 16 hours ago[Dwarkesh Podcast](http://www.youtube.com/playlist?list=PLd7-bHaQwnthaNDpZ32TtYONGVk95-fhF) 80,828 views • Apr 29, 2026 • Dwarkesh Podcast Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and served. It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there - it’s really worth it. …...more ...more ## Chapters View all [![Image 8](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How batch size affects token cost and speed #### How batch size affects token cost and speed 0:00](http://www.youtube.com/watch?v=xmkSf5IS-zw&pp=0gcJCREDDuyUWbzu) #### [How batch size affects token cost and speed](http://www.youtube.com/watch?v=xmkSf5IS-zw&pp=0gcJCREDDuyUWbzu) 0:00 [![Image 9](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How MoE models are laid out across GPU racks #### How MoE models are laid out across GPU racks 31:59](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=1919s) #### [How MoE models are laid out across GPU racks](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=1919s) 31:59 [![Image 10](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How pipeline parallelism spreads model layers across racks #### How pipeline parallelism spreads model layers across racks 47:02](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=2822s) #### [How pipeline parallelism spreads model layers across racks](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=2822s) 47:02 [![Image 11](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Why Ilya said, “As we now know, pipelining is not wise.” #### Why Ilya said, “As we now know, pipelining is not wise.” 1:03:27](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=3807s) #### [Why Ilya said, “As we now know, pipelining is not wise.”](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=3807s) 1:03:27 [![Image 12](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Because of RL, models may be 100x over-trained beyond Chinchilla-optimal #### Because of RL, models may be 100x over-trained beyond Chinchilla-optimal 1:18:49](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=4729s&pp=0gcJCREDDuyUWbzu) #### [Because of RL, models may be 100x over-trained beyond Chinchilla-optimal](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=4729s&pp=0gcJCREDDuyUWbzu) 1:18:49 [![Image 13](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Deducing long context memory costs from API pricing #### Deducing long context memory costs from API pricing 1:32:52](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=5572s) #### [Deducing long context memory costs from API pricing](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=5572s) 1:32:52 [![Image 14](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Convergent evolution between neural nets and cryptography #### Convergent evolution between neural nets and cryptography 2:03:52](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=7432s) #### [Convergent evolution between neural nets and cryptography](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=7432s) 2:03:52 Explore the podcast [![Image 15](http://www.youtube.com/watch?v=xmkSf5IS-zw)126 episodes 126 episodes](http://www.youtube.com/playlist?list=PLd7-bHaQwnthaNDpZ32TtYONGVk95-fhF) Dwarkesh Podcast Dwarkesh Patel [![Image 16](http://www.youtube.com/watch?v=xmkSf5IS-zw) Podcasts](http://www.youtube.com/podcasts) Transcript Follow along using the transcript. Show transcript [![Image 17](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### Dwarkesh Patel 1.29M subscribers](http://www.youtube.com/@DwarkeshPatel) [Videos](http://www.youtube.com/channel/UCXl4i9dYBrFOabk0xGmbkRA/videos)[About](http://www.youtube.com/channel/UCXl4i9dYBrFOabk0xGmbkRA/about)[![Image 18](http://www.youtube.com/watch?v=xmkSf5IS-zw) Twitter](https://www.youtube.com/redirect?event=Watch_SD_EP&redir_token=QUFFLUhqbGdHRXNfMjVnd1VOdEJmQkRidjZEaGRneGVVZ3xBQ3Jtc0trV3JGcHN0M3p2Um9ZdGNuWGpZZmdWZ1F2NWxNNjFETUNDVEZPZ0NOaWVmcVMxNmtZYjUtNFRyVExneVpVeEgwZ2RVQW1wVVJydUl6VW95U3d4TVBwbVY3NTlJR1RRUGdJVVpZTzRMYzJXUFpFNFpnWQ&q=https%3A%2F%2Ftwitter.com%2Fdwarkesh_sp)[![Image 19](http://www.youtube.com/watch?v=xmkSf5IS-zw) Clips](https://www.youtube.com/@DwarkeshClips) Show less [](http://www.youtube.com/playlist?list=PLd7-bHaQwnthaNDpZ32TtYONGVk95-fhF)[Dwarkesh Podcast](http://www.youtube.com/playlist?list=PLd7-bHaQwnthaNDpZ32TtYONGVk95-fhF) # How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope 80,828 views Apr 29, 2026 3.4K Share Save Download Download ## 209 Comments Sort comments Sort by [Top Show featured comments](http://www.youtube.com/watch?v=xmkSf5IS-zw)[Newest Show recent comments, including potential spam](http://www.youtube.com/watch?v=xmkSf5IS-zw) ![Image 20: Default profile photo](https://yt3.ggpht.com/a/default-user=s48-c-k-c0x00ffffff-no-rj) Add a comment... ![Image 21](https://yt3.ggpht.com/lG-z7sTfhFIW2Ne1oXMHvXMXyZSaA02_I17gUel0GAEj7OypsSHQ7PE91Vp4bTbpm3PTIAWJdko=s88-c-k-c0x00ffffff-no-rj) Pinned by @DwarkeshPatel ### [@DwarkeshPatel](http://www.youtube.com/@DwarkeshPatel) [@DwarkeshPatel @DwarkeshPatel](http://www.youtube.com/@DwarkeshPatel)[14 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=Ugz-hpixIQcBHmJLg4R4AaABAg) Wrote up some flashcards and practice problems to help myself retain what Reiner taught. Hope it's helpful to you too! [https://reiner-flashcards.vercel.app](https://www.youtube.com/redirect?event=comments&redir_token=QUFFLUhqbGYyX3dzdmEtSXpKWk8zTXh0STNOWEctaGhRUXxBQ3Jtc0trcG1kTEIzQ1dRdjVSUVRIODBPY0NHWEMtN0hFd2NOOTVaQ3MwczRoM1pPcnJrb2dfQmRQRVdxXzlxZlF6cklUOFN6SmZ3Mmwxd2VKWXA3QUZPN0UtV1FfSFVJSVlrSzhtWlpldlVQbkItOXotNVUyaw&q=https%3A%2F%2Freiner-flashcards.vercel.app%2F) Show less Read more Like 50 Dislike Reply 2 replies Hide replies 2 replies ![Image 22](https://yt3.ggpht.com/ytc/AIdro_nCPssRuNx9H2TLciGO-EEf7Z1G79v7eWzZmXK8SoluRvop=s88-c-k-c0x00ffffff-no-rj) ### [@BLAISEDAHL96](http://www.youtube.com/@BLAISEDAHL96) [15 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgwnCa0B0cIc1ASb-vl4AaABAg) Yep, this definitely needs to be the format moving forward with guests that care to instruct something. This is a great public service. Show less Read more Like 308 Dislike Reply 1 reply Hide replies 1 reply ![Image 23](https://yt3.ggpht.com/7v7MvdmZVrWvtsV7AZpcUrF8nL6EZhARrNoIL1pDa4-e13OxHK46cPT3MY2Se21mlQ6WB8Zo=s88-c-k-c0x00ffffff-no-rj) ### [@WillzMaster85](http://www.youtube.com/@WillzMaster85) [15 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgxQ3zKHullOzEIrFgp4AaABAg) this new format will be a complete gamechanger; a simple yet genius move dwarkesh, good work! Show less Read more Like 156 Dislike Reply ![Image 24](https://yt3.ggpht.com/ytc/AIdro_mdI7mNS60VQDXUqvA-AwiOL8U3GpJ-8gBHEGDSQIsyuFU=s88-c-k-c0x00ffffff-no-rj) ### [@fernandodutra3788](http://www.youtube.com/@fernandodutra3788) [12 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgzdXID5uBkePFwWMYV4AaABAg&pp=0gcJCSIANpG00pGi) So nice to see a mainstream podcast (1.3M subs) spend 2+h discussing technical state of the art. Not just “let’s explain what 1+1 is so the audience can follow”, but actually just going with the flow. Appreciated! Show less Read more Like 60 Dislike Reply ![Image 25](https://yt3.ggpht.com/ytc/AIdro_ldUXF8b4ZkF0mOPcQBYNHz4Fl5YcRkV2OOlVzNOW7fxnI=s88-c-k-c0x00ffffff-no-rj) ### [@manushsanchela1109](http://www.youtube.com/@manushsanchela1109) [15 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=Ugxs1pioZEtQxEZ0hpV4AaABAg&pp=0gcJCSIANpG00pGi) This is the best format I have seen on YT. Please keep going Show less Read more Like 46 Dislike Reply ![Image 26](https://yt3.ggpht.com/3vBBk4FChDJHDwzdq65nj4Cwpq_EKeirib6sWB_78AQYXvBNnmfa69ISC7YivqPueX3Cf-QLog=s88-c-k-c0x00ffffff-no-rj) ### [@darkflow_null](http://www.youtube.com/@darkflow_null) [15 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=Ugxy2yS4zos2kkw69BJ4AaABAg) Absolutely love the new format Show less Read more Like 42 Dislike Reply ![Image 27](https://yt3.ggpht.com/navg5xEVZYCoBdkQ42DjNj18VdCuBFvMkLoaWGOBnHa5e0Jnd3XVA3XigUyMpHN77vgeJ__kOA=s88-c-k-c0x00ffffff-no-rj) ### [@FedericoLebrón](http://www.youtube.com/@FedericoLebr%C3%B3n) [14 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgwNDwa7Rgu3a4G2J6J4AaABAg) Reiner and a blackboard? I've never clicked on anything faster in my life. What a treat!!!! Show less Read more Like 21 Dislike Reply ![Image 28](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@JunaidHasan23](http://www.youtube.com/@JunaidHasan23) [10 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgzAX2XEuBf-hB6MVP94AaABAg) Plot is incorrect at [9:29](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=569s) when he adds the two curves for the weight fetch + KV fetch. The line has to be parallel to KV fetch. Show less Read more Like 17 Dislike Reply 1 reply Hide replies 1 reply ![Image 29](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@djelvis1](http://www.youtube.com/@djelvis1) [14 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgwAaQCmpjEjSurs56t4AaABAg) DUDE. THIS IS FUCKING SICK. WHAT THE FUCK?! Show less Read more Like 20 Dislike Reply ![Image 30](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@BLAISEDAHL96](http://www.youtube.com/@BLAISEDAHL96) [15 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgyVLWoWDew_lL3aDut4AaABAg) This looks phenomenal! One note would be to get some more fill lighting so you guys aren’t completely shaded out. Show less Read more Like 8 Dislike Reply ![Image 31](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@SudhamsuSharmaneodonly](http://www.youtube.com/@SudhamsuSharmaneodonly) [10 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgwcwFF-3m9Qeu6ZKnN4AaABAg) I thought thumbnail was bait thinking there wont be any board, but wow! Interesting format Show less Read more Like 6 Dislike Reply ![Image 32](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@chyldstudios](http://www.youtube.com/@chyldstudios) [49 minutes ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=Ugz6p8ZLXKjYjS9-Uhl4AaABAg) more videos with math + whiteboard please! Show less Read more Like 2 Dislike Reply ![Image 33](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@ViralGrowth2](http://www.youtube.com/@ViralGrowth2) [10 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=Ugxyza8pI4BROqGagkd4AaABAg) With such an influx of people new to AI, this is a perfect format for learning Show less Read more Like 3 Dislike Reply ![Image 34](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@Zvnasif](http://www.youtube.com/@Zvnasif) [16 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=Ugw8AwtrUgLu-rblZoV4AaABAg&pp=0gcJCSIANpG00pGi) The best thing you can do with a Podcast - deep dive on a concept with sketches Show less Read more Like 8 Dislike Reply ![Image 35](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@alex-ca-p3i](http://www.youtube.com/@alex-ca-p3i) [16 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgySfXTq45XKrAeSgjd4AaABAg&pp=0gcJCSIANpG00pGi) thanks for inviting the Pope Show less Read more Like 9 Dislike Reply ![Image 36](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@Rick-134uy5](http://www.youtube.com/@Rick-134uy5) [15 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgzZiPLvsDzAphlE6F14AaABAg) Great format with blackboard. Love it! Do more of it Show less Read more Like 8 Dislike Reply ![Image 37](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@gadwal](http://www.youtube.com/@gadwal) [4 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=Ugxo-nng-VVMZ82uT7J4AaABAg) SOTA engineers taking the talk to the blackboard is such a fresh feel. Show less Read more Like 1 Dislike Reply ![Image 38](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@laslog](http://www.youtube.com/@laslog) [3 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=Ugwx-am2U0TxMGY8S6l4AaABAg) Loving the blackboard format, crazy good!!! Show less Read more Like Dislike Reply ![Image 39](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@JyotishmanHazarika-hs3ku](http://www.youtube.com/@JyotishmanHazarika-hs3ku) [16 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgzegQ_vIuNtxVr960V4AaABAg) Saved! Thanks for bringing more and more interesting personalities with deep specializations. Show less Read more Like 4 Dislike Reply ![Image 40](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [@laycookie-f6i](http://www.youtube.com/@laycookie-f6i) [11 hours ago](http://www.youtube.com/watch?v=xmkSf5IS-zw&lc=UgxGZUhMJ7ugKpQH32p4AaABAg) YouTube was not made for this type of quality video. [2:29](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=149s) you can just see the bitrate die on the background even though pretty much nothing is moving. Show less Read more Like 2 Dislike Reply ## Comments 209 [Top Show featured comments](http://www.youtube.com/watch?v=xmkSf5IS-zw)[Newest Show recent comments, including potential spam](http://www.youtube.com/watch?v=xmkSf5IS-zw) ## In this video Chapters Transcript ## Chapters [![Image 41](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How batch size affects token cost and speed #### How batch size affects token cost and speed 0:00](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### [How batch size affects token cost and speed](http://www.youtube.com/watch?v=xmkSf5IS-zw) 0:00 [![Image 42](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How MoE models are laid out across GPU racks #### How MoE models are laid out across GPU racks 31:59](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=1919s) #### [How MoE models are laid out across GPU racks](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=1919s) 31:59 [![Image 43](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How pipeline parallelism spreads model layers across racks #### How pipeline parallelism spreads model layers across racks 47:02](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=2822s) #### [How pipeline parallelism spreads model layers across racks](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=2822s) 47:02 [![Image 44](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Why Ilya said, “As we now know, pipelining is not wise.” #### Why Ilya said, “As we now know, pipelining is not wise.” 1:03:27](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=3807s&pp=0gcJCREDDuyUWbzu) #### [Why Ilya said, “As we now know, pipelining is not wise.”](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=3807s&pp=0gcJCREDDuyUWbzu) 1:03:27 [![Image 45](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Because of RL, models may be 100x over-trained beyond Chinchilla-optimal #### Because of RL, models may be 100x over-trained beyond Chinchilla-optimal 1:18:49](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=4729s) #### [Because of RL, models may be 100x over-trained beyond Chinchilla-optimal](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=4729s) 1:18:49 [![Image 46](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Deducing long context memory costs from API pricing #### Deducing long context memory costs from API pricing 1:32:52](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=5572s) #### [Deducing long context memory costs from API pricing](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=5572s) 1:32:52 [![Image 47](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Convergent evolution between neural nets and cryptography #### Convergent evolution between neural nets and cryptography 2:03:52](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=7432s) #### [Convergent evolution between neural nets and cryptography](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=7432s) 2:03:52 Sync to video time ## Description How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope 3.4K Likes 80,828 Views 16h Ago Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and served. It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there - it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him. Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture. Wrote up some flashcards and practice problems to help myself retain what Reiner taught. Hope it's helpful to you too! [https://reiner-flashcards.vercel.app/](https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbmdGbXhIWHNVY0JpY1NTdGJUbDRLanVBQXZPQXxBQ3Jtc0ttaGR3T29UdUw3TlV0SVdBcDNvbUNLcENreERSbmRWY2JDLUVkX3lCWndWSmg5LXU5dnhDeGlJeXk0RnlKbXhzbTQ0a3VZbDF2MThHVndobENZZE9WLW9BcVVtVnBFNUZDbUgxY1NPeXpJVUJpdXh0SQ&q=https%3A%2F%2Freiner-flashcards.vercel.app%2F&v=xmkSf5IS-zw) Download markdown of transcript here to chat with an LLM: [https://gist.github.com/dwarkeshsp/79...](https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbjNWT0tsaUN4NkVvSlJGcHdlY2JBa2FBV3NlQXxBQ3Jtc0ttS2RsbmhVU0EzT3F2dE9oQ3U0YU9MWFI5c1E1QlFFMEU5ZktPcFk4RDN6RHpvN2NHVmRfZnhzcnhxVmFlZ3pockhwSGZXdjgxNllfVVJCdzdlaDdJaE45WEhvN1NjMHBEdmhadlNpSnBNeGRXMUlBYw&q=https%3A%2F%2Fgist.github.com%2Fdwarkeshsp%2F79100f0fdeed69d76241903bb0604dbe&v=xmkSf5IS-zw)[0:00:00](http://www.youtube.com/watch?v=xmkSf5IS-zw) – How batch size affects token cost and speed [0:31:59](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=1919s) – How MoE models are laid out across GPU racks [0:47:02](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=2822s) – How pipeline parallelism spreads model layers across racks [1:03:27](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=3807s) – Why Ilya said, “As we now know, pipelining is not wise.” [1:18:49](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=4729s) – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal [1:32:52](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=5572s) – Deducing long context memory costs from API pricing [2:03:52](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=7432s) – Convergent evolution between neural nets and cryptography…...more ...more Show less ## Chapters View all [![Image 48](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How batch size affects token cost and speed #### How batch size affects token cost and speed 0:00](http://www.youtube.com/watch?v=xmkSf5IS-zw&pp=0gcJCREDDuyUWbzu) #### [How batch size affects token cost and speed](http://www.youtube.com/watch?v=xmkSf5IS-zw&pp=0gcJCREDDuyUWbzu) 0:00 [![Image 49](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How MoE models are laid out across GPU racks #### How MoE models are laid out across GPU racks 31:59](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=1919s) #### [How MoE models are laid out across GPU racks](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=1919s) 31:59 [![Image 50](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### How pipeline parallelism spreads model layers across racks #### How pipeline parallelism spreads model layers across racks 47:02](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=2822s) #### [How pipeline parallelism spreads model layers across racks](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=2822s) 47:02 [![Image 51](http://www.youtube.com/watch?v=xmkSf5IS-zw) #### Why Ilya said, “As we now know, pipelining is not wise.” #### Why Ilya said, “As we now know, pipelining is not wise.” 1:03:27](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=3807s) #### [Why Ilya said, “As we now know, pipelining is not wise.”](http://www.youtube.com/watch?v=xmkSf5IS-zw&t=3807s) 1:03:27 Explore the podcast [![Image 52](http://www.youtube.com/watch?v=xmkSf5IS-zw)126 episodes 126 episodes](http://www.youtube.com/playlist?list=PLd7-bHaQwnthaNDpZ32TtYONGVk95-fhF) Dwarkesh Podcast Dwarkesh Patel [![Image 53](http://www.youtube.com/watch?v=xmkSf5IS-zw) Podcasts](http://www.youtube.com/podcasts) Transcript Follow along using the transcript. Show transcript [![Image 54](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### Dwarkesh Patel 1.29M subscribers](http://www.youtube.com/@DwarkeshPatel) [Videos](http://www.youtube.com/channel/UCXl4i9dYBrFOabk0xGmbkRA/videos)[About](http://www.youtube.com/channel/UCXl4i9dYBrFOabk0xGmbkRA/about)[![Image 55](http://www.youtube.com/watch?v=xmkSf5IS-zw) Twitter](https://www.youtube.com/redirect?event=Watch_SD_EP&redir_token=QUFFLUhqbGdHRXNfMjVnd1VOdEJmQkRidjZEaGRneGVVZ3xBQ3Jtc0trV3JGcHN0M3p2Um9ZdGNuWGpZZmdWZ1F2NWxNNjFETUNDVEZPZ0NOaWVmcVMxNmtZYjUtNFRyVExneVpVeEgwZ2RVQW1wVVJydUl6VW95U3d4TVBwbVY3NTlJR1RRUGdJVVpZTzRMYzJXUFpFNFpnWQ&q=https%3A%2F%2Ftwitter.com%2Fdwarkesh_sp)[![Image 56](http://www.youtube.com/watch?v=xmkSf5IS-zw) Clips](https://www.youtube.com/@DwarkeshClips) ## Transcript NaN / NaN [![Image 57](https://i.ytimg.com/vi/96jN2OCOfLs/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLBZdp1cX62GE_0F_7p8NNyecd0HIQ) 29:49](http://www.youtube.com/watch?v=96jN2OCOfLs&pp=ugUEEgJlbg%3D%3D) ![Image 58](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Andrej Karpathy: From Vibe Coding to Agentic Engineering](http://www.youtube.com/watch?v=96jN2OCOfLs&pp=ugUEEgJlbg%3D%3D) Sequoia Capital 151K views • 18 hours ago New [![Image 59](https://i.ytimg.com/vi/JNyuX1zoOgU/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLDn0yBys-Of1HHHC6hqYyutUUpWPg) 40:57](http://www.youtube.com/watch?v=JNyuX1zoOgU&pp=ugUHEgVlbi1VUw%3D%3D) ![Image 60](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Demis Hassabis: Agents, AGI & The Next Big Scientific Breakthrough](http://www.youtube.com/watch?v=JNyuX1zoOgU&pp=ugUHEgVlbi1VUw%3D%3D) Y Combinator 50K views • 19 hours ago New [![Image 61](https://i.ytimg.com/vi/6eBSHbLKuN0/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLDLIzYUE8oh_gGWHezmzFwPxYzRxg) 28:07](http://www.youtube.com/watch?v=6eBSHbLKuN0) ![Image 62](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Mastering Claude Code in 30 minutes](http://www.youtube.com/watch?v=6eBSHbLKuN0) Anthropic 1.1M views • Streamed 11 months ago [![Image 63](https://i.ytimg.com/vi/Hrbq66XqtCo/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLDeSxvUHCrfe3plRM-i9XQ-PwTMLA) 1:43:13](http://www.youtube.com/watch?v=Hrbq66XqtCo&pp=ugUEEgJlbg%3D%3D) ![Image 64](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Jensen Huang – Will Nvidia’s moat persist?](http://www.youtube.com/watch?v=Hrbq66XqtCo&pp=ugUEEgJlbg%3D%3D) Dwarkesh Patel 733K views • 2 weeks ago [![Image 65](https://i.ytimg.com/vi/Ykvf3MunGf8/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLCjq_ZUrhyQ0Cmk9x71_5cfH7QjSw) 29:43](http://www.youtube.com/watch?v=Ykvf3MunGf8) ![Image 66](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [AI Chatbots: Last Week Tonight with John Oliver (HBO)](http://www.youtube.com/watch?v=Ykvf3MunGf8) LastWeekTonight 2.8M views • 3 days ago New [![Image 67](https://i.ytimg.com/vi/BYXbuik3dgA/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLBZFxmmXun2Z8OPxm3eStZWwiIW3w) 2:49:46](http://www.youtube.com/watch?v=BYXbuik3dgA&pp=ugUEEgJlbg%3D%3D) ![Image 68](http://www.youtube.com/watch?v=xmkSf5IS-zw) ![Image 69](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Elon Musk – "In 36 months, the cheapest place to put AI will be space”](http://www.youtube.com/watch?v=BYXbuik3dgA&pp=ugUEEgJlbg%3D%3D) Dwarkesh Patel and Stripe 1.3M views • 2 months ago [![Image 70](https://i.ytimg.com/vi/1ySBxK7viNs/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLCvQX_lfbZlJIiI5Bg6G8Ni3YHGrw) 58:20](http://www.youtube.com/watch?v=1ySBxK7viNs&pp=0gcJCd8KAYcqIYzv) ![Image 71](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Sam Altman in conversation with Patrick Collison](http://www.youtube.com/watch?v=1ySBxK7viNs&pp=0gcJCd8KAYcqIYzv) Stripe 6.3K views • 6 hours ago New [![Image 72](https://i.ytimg.com/vi/ksn5yrsC3Wg/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLDtzKExvS0-j_o3VUrTPEu9uoWQNA) 33:25](http://www.youtube.com/watch?v=ksn5yrsC3Wg) ![Image 73](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [The disaster I never imagined having to worry about](http://www.youtube.com/watch?v=ksn5yrsC3Wg) Veritasium 1.2M views • 13 hours ago New [![Image 74](https://i.ytimg.com/vi/AFpeWo1GTeg/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLDlf3atwgHS743ZTvHpzCZa2UYpVA) 26:52](http://www.youtube.com/watch?v=AFpeWo1GTeg&pp=ugUEEgJlbtIHCQnfCgGHKiGM7w%3D%3D) ![Image 75](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Demis Hassabis: We're Three Quarters of the Way to AGI](http://www.youtube.com/watch?v=AFpeWo1GTeg&pp=ugUEEgJlbtIHCQnfCgGHKiGM7w%3D%3D) Sequoia Capital 47K views • 18 hours ago New [![Image 76](https://i.ytimg.com/vi/n5f51gtuGHE/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLBsePwQGhaQuqhA1yzn9CYaxmlNhA) 1:33:56](http://www.youtube.com/watch?v=n5f51gtuGHE&pp=ugUHEgVlbi1VUw%3D%3D) ![Image 77](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Building Pi, and what makes self-modifying software so fascinating](http://www.youtube.com/watch?v=n5f51gtuGHE&pp=ugUHEgVlbi1VUw%3D%3D) The Pragmatic Engineer 16K views • 19 hours ago New [![Image 78](https://i.ytimg.com/vi/UQZooauU-FQ/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLDOO5tFlmgXqW3NYzCXIw_mUyXBhw) 24:02](http://www.youtube.com/watch?v=UQZooauU-FQ) ![Image 79](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Humanoid Robots and the Gap Between Hype and Reality | Bloomberg Primer](http://www.youtube.com/watch?v=UQZooauU-FQ) Bloomberg Originals 101K views • 1 day ago New [![Image 80](https://i.ytimg.com/vi/n1E9IZfvGMA/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLBt3QmiYigniDt_qjt_-TohM3W_uQ) 2:22:20](http://www.youtube.com/watch?v=n1E9IZfvGMA&pp=ugUEEgJlbg%3D%3D) ![Image 81](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Dario Amodei — “We are near the end of the exponential”](http://www.youtube.com/watch?v=n1E9IZfvGMA&pp=ugUEEgJlbg%3D%3D) Dwarkesh Patel 1M views • 2 months ago [![Image 82](https://i.ytimg.com/vi/9aIYhjeYxzM/hqdefault.jpg?sqp=-oaymwEmCKgBEF5IWvKriqkDGQgBFQAAiEIYAdgBAeIBCggYEAIYBjgBQAE=&rs=AOn4CLD7JW3ictJbtqzMwPmPNalwvgulug) 32:34](http://www.youtube.com/watch?v=9aIYhjeYxzM&pp=ugUEEgJlbg%3D%3D) ![Image 83](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [GPT-5.5 vs Claude vs Gemini: The Real Difference Nobody's Talking About](http://www.youtube.com/watch?v=9aIYhjeYxzM&pp=ugUEEgJlbg%3D%3D) AI News & Strategy Daily | Nate B Jones 77K views • 1 day ago New [![Image 84](http://www.youtube.com/watch?v=xmkSf5IS-zw) 11:09](http://www.youtube.com/watch?v=i9TvUGeTltE) ![Image 85](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [My Take on The New Apple](http://www.youtube.com/watch?v=i9TvUGeTltE) Marques Brownlee 1.1M views • 10 hours ago New [![Image 86](http://www.youtube.com/watch?v=xmkSf5IS-zw) 1:05:11](http://www.youtube.com/watch?v=C0gErQtnNFE) ![Image 87](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [The Hardest Problem AI Ever Solved, with Google DeepMind CEO](http://www.youtube.com/watch?v=C0gErQtnNFE) Cleo Abram 1.1M views • 3 weeks ago [![Image 88](http://www.youtube.com/watch?v=xmkSf5IS-zw) 17:53](http://www.youtube.com/watch?v=jfiBDuUVc-A&pp=ugUEEgJlbg%3D%3D) ![Image 89](http://www.youtube.com/watch?v=xmkSf5IS-zw) ![Image 90](http://www.youtube.com/watch?v=xmkSf5IS-zw) ![Image 91](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [How to pull yourself out of a cheap dopamine spiral | Mark Manson](http://www.youtube.com/watch?v=jfiBDuUVc-A&pp=ugUEEgJlbg%3D%3D) Big Think Clips and 2 more 183K views • 1 day ago New [![Image 92](http://www.youtube.com/watch?v=xmkSf5IS-zw) 1:36:30](http://www.youtube.com/watch?v=-QFHIoCo-Ko&pp=ugUEEgJlbg%3D%3D) ![Image 93](http://www.youtube.com/watch?v=xmkSf5IS-zw) ![Image 94](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Full Walkthrough: Workflow for AI Coding — Matt Pocock](http://www.youtube.com/watch?v=-QFHIoCo-Ko&pp=ugUEEgJlbg%3D%3D) AI Engineer and Matt Pocock 217K views • 5 days ago New [![Image 95](http://www.youtube.com/watch?v=xmkSf5IS-zw) 1:23:44](http://www.youtube.com/watch?v=Q8Fkpi18QXU&pp=ugUEEgJlbg%3D%3D) ![Image 96](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Terence Tao – How the world’s top mathematician uses AI](http://www.youtube.com/watch?v=Q8Fkpi18QXU&pp=ugUEEgJlbg%3D%3D) Dwarkesh Patel 400K views • 1 month ago [![Image 97](http://www.youtube.com/watch?v=xmkSf5IS-zw) 18:17](http://www.youtube.com/watch?v=9Qh4bHqQlnQ&pp=0gcJCd8KAYcqIYzv) ![Image 98](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [How to Think So Clearly People Assume You’re A Genius](http://www.youtube.com/watch?v=9Qh4bHqQlnQ&pp=0gcJCd8KAYcqIYzv) Justin Sung 168K views • 6 days ago New [![Image 99](http://www.youtube.com/watch?v=xmkSf5IS-zw) 10:41](http://www.youtube.com/watch?v=cqGfyXF6_3w&pp=ugUEEgJlbg%3D%3D) ![Image 100](http://www.youtube.com/watch?v=xmkSf5IS-zw) ### [Hey Anthropic, That Looks Illegal?](http://www.youtube.com/watch?v=cqGfyXF6_3w&pp=ugUEEgJlbg%3D%3D) Tim Carambat 17K views • 11 hours ago New Show more [](http://www.youtube.com/watch?v=xmkSf5IS-zw)