从零构建推理模型:完整视频课程

过去两年,我在 Ahead of AI 上写了很多关于推理模型的内容,包括最近一篇关于当代 LLM 中控制推理努力程度机制的文章。推理模型如今依然同样重要。事实上,当前这一代 LLM 几乎全部都是推理模型,或者提供推理模式。
如果你对推理模型的工作原理感兴趣,包括推理扩展、强化学习和蒸馏,可以看看我最近写的《从零构建推理模型》(Build Reasoning Model (From Scratch)),它用 PyTorch 从底层实现了这些内容。以我的经验来看,这仍然是理解和学习事物真正运作方式的最佳途径。
最近几个月,我还专门坐下来录制了想与大家分享的配套讲座。所以,下面你会看到 8 个从零构建推理模型的编码视频(共 11 小时内容)。
我们从预训练的 LLM 出发,用 Python 和 PyTorch 实现评估和改进它的方法。所以,如果你喜欢我去年开设的 Coding LLMs from the Ground Up 课程,这基本上就是它的续集(不过你也可以在没有先修该课程的情况下直接观看)。
我目前正在短暂休息,下个月会回来,带来更多令人兴奋的 LLM 研究综述。在此期间,希望你能享受跟着一起编码的乐趣!
另外补充一点,由于经常有人问我推理模型与当今 AI 智能体的关系,在开始看视频之前,我先用一小节来谈谈这个问题。
推理模型本质上是对传统 LLM 的一次升级,即通过带可验证奖励的强化学习加入更多后训练(也可选地结合推理扩展技术),使其更有能力解决更具挑战性的问题。传统上,推理模型是为数学和代码而开发的,但这些能力也可以泛化到其他问题领域。
于是,推理模型充当智能体框架(agent harness)内部的引擎,而智能体框架是围绕 LLM/推理模型构建的软件或工具框架,用来应对更具挑战性的问题(更大的编码任务、旅行预订,以及各类个人助理任务)。该框架的任务是进行复杂的上下文管理,并让 LLM 能够访问各种工具。
图 1:用引擎比喻传统 LLM、推理模型与智能体框架。图出自我的《Components of a Coding Agent》一文。
在上图所示的这个比喻中,模型是引擎,而智能体框架则加装了各种机械结构,让我们能把它用于各种用途(相比简单的聊天机器人界面是一种升级)。智能体既可以使用传统 LLM,也可以使用推理模型,但推理模型是更典型的选择,因为它们能力更强。(如上所述,当前大多数世代的 LLM 都是带有可调推理强度设置的推理模型。)
因此,归根结底,作为“引擎”的推理模型仍是理解现代 LLM 智能体的重要一环。
上文这些主题所配套的资源包括:
《Build a Large Language Model (From Scratch)》一书及视频
《Build A Reasoning Model (From Scratch)》一书(以及下方视频)
Substack 上《The Components of a Coding Agent》一文
以下视频聚焦于开发能够支撑此类系统的推理模型。
“从零构建推理”系列视频以一个较小的预训练 Qwen3 模型为基础,我们自己在 Python 和 PyTorch 中实现各种推理方法。从预训练模型出发,可以让我们聚焦于真正有趣的部分:评估、推理时扩展(inference-time scaling)、强化学习和蒸馏。
关于前置要求,我建议你对 Python 和 PyTorch 比较熟悉。我把推理部分的内容设计为独立内容,因此此前《Build A Large Language Model (From Scratch)》的内容是推荐而非硬性要求。
在算力资源方面,早期的文本生成和评估示例可以在 CPU 上跑通。对于后续实验,尤其是训练,有 GPU 会让工作实用得多。第一个视频涵盖了环境搭建以及更多建议;代码仓库中也包含硬件和环境方面的推荐。
图 2:所涵盖主题的概览。
这些视频是我所著《Build a Reasoning Model (From Scratch)》一书的配套内容。如果你想更详细地、按自己的节奏阅读讲解,并完成练习等,我推荐把这本书作为配套材料。
若想获得最深入的学习体验,我的建议是:先读一章,配合对应视频动手跑一遍代码,然后再尝试练习。
这本书可在 Manning 和 Amazon 购买,纸质版是全彩印刷的!
图 3:《Build a Reasoning Model (From Scratch)》纸质书中的随机摘录。
感谢每一位购买本书或把它推荐给朋友的人。你们的支持帮助我持续创作这些内容。
在进入下方主要视频之前,最后再简单提一句:如果想看更简短、更偏概念性的概览,我也推荐我在 LinkedIn Learning 上的 AI Reasoning Models 课程(通过那里的链接访问时,24 小时内应该可以免费学习,而且无需 LinkedIn Premium 订阅)。
该课程大约 90 分钟,讲解推理模型如何工作,涵盖推理时扩展、强化学习和蒸馏背后的主要思路。如果你只想了解核心思想、不想跟着下面的完整实现动手做,这是一个不错的可选预习材料。
进入本文的主体部分,以下是八个主要讲座视频。
第一个视频首先解释传统 LLM、推理模型与智能体之间的关系。接着,视频的主要主题是搭建编程环境,包括安装 Python 和 PyTorch 依赖项(使用 uv)等。我还会讨论硬件选择以及常见的环境配置问题。
涵盖主题
常规 LLM、推理模型与智能体框架之间的关系
关于通过代码实现来学习的好处
本视频系列的范围
安装 Python 依赖项
选择硬件并检查 CUDA 或 Apple Silicon 支持
使用 VS Code 连接远程 GPU 并排查环境问题
观看视频 1 · 设置说明
在后续视频中引入推理方法之前,我们需要一个可用的基础模型。我们不从头训练它,因为那会是另一整本书的内容(参见《Build A Large Language Model (From Scratch)》),而是从一个预训练模型开始。
在这里,我们加载 Qwen3 0.6B 模型的预训练权重。接着,我们通过流式文本生成器实现文本生成流程,并加入 KV 缓存和模型编译,让文本生成更高效。
这些实现为我们提供了基线和基础组件,我们将在后续视频中对它们进行评估和改进。
涵盖的主题
将文本编码为 Token 并再解码回文本
下载 Qwen3 权重并在选定的计算设备上加载模型
讲解张量形状、logits 和下一 Token 预测
实现贪心解码与流式文本生成
处理序列结束 Token 并测量生成速度
添加 KV 缓存以复用注意力键和值
将 KV 缓存与 torch.compile 结合,并比较硬件性能
观看视频 2 · 第 2 章代码
在本讲中,我们逐步构建一个数学答案验证器,从答案提取到归一化,再到检查数学等价性。随后我们用它来在 MATH-500 上评估模型。
我们使用该验证器来评估基础模型在 MATH-500 评估数据集上的表现。
不过,该验证器在本系列后续内容中还有第二项任务:为可验证奖励强化学习(RLVR)提供奖励信号。
涵盖主题
验证器简介,以及它们如何融入 LLM 评估方法(其他方法在本书附录中介绍)
复用答案验证作为 RLVR 的奖励信号(第 6 章将进一步展开)
验证器流程,包括提取最终答案、对答案进行归一化以及检查数学等价性
为 MATH-500 构建答案评分器和评估循环
在 MATH-500 上比较基础模型与推理模型
观看视频 3 · 第 3 章代码
现在我们探讨在保持模型权重不变的前提下改进答案的方法。但在实现具体的推理扩展技术之前,我们先实现从同一提示词生成多样化答案的方法(这是 ChatGPT 等专有 LLM 聊天机器人的标准做法)。
具体来说,我们实现温度缩放、多项式采样和 top-p 过滤来生成多样化的回答。然后通过自洽性(self-consistency)把多个回答组合起来,对其最终答案进行多数投票。
我们还考察答案准确率与生成更多候选答案所需的额外计算量之间的权衡。
涵盖主题
比较额外算力在训练和推理阶段如何提升模型性能
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Building Reasoning Models from the Ground Up: A Full Video Course)
I have written a lot about reasoning models here over the last two years on Ahead of AI, including a recent article on mechanisms to control reasoning effort levels in contemporary LLMs. Reasoning models are still as relevant as ever. In fact, almost all current-generation LLMs today are reasoning models or offer a reasoning mode.
For those who are interested in how reasoning models work, including inference scaling, reinforcement learning, and distillation, I wrote my recent Build Reasoning Model (From Scratch), which implements these from the ground up in PyTorch, which, in my experience, is still the best way to understand and learn how things really work.
Now, in the last few months, I also sat down and recorded accompanying lectures I wanted to share with you. So, below, you will find eight reasoning model-from-scratch coding videos (11 hours of content).
We start with a pre-trained LLM and implement methods to evaluate and improve it in Python and PyTorch. So, if you enjoyed my Coding LLMs from the Ground Up course last year, this is essentially the sequel (but you can also watch it without the prior course as a prerequisite).
I’m currently taking a short break and will be back soon with more exciting LLM research roundups next month. In the meantime, I hope you enjoy coding along!
How LLMs, reasoning models, and agents relate and fit together
As a small side note, and since I often get asked how reasoning models relate to today’s AI agents, here’s a short section on this before getting started with the videos.
A reasoning model is essentially an upgrade from a conventional LLM, that is, adding more post-training via reinforcement learning with verifiable rewards (and optionally inference scaling techniques) to make it more capable of solving more challenging problems. Classically, reasoning models are developed for math and code, but the capabilities can generalize to other problem domains as well.
A reasoning model then functions as the engine inside an agent harness, where the agent harness is a software or tooling framework around the LLM/reasoning model to tackle even more challenging problems (bigger coding tasks, travel booking, and all other sorts of personal assistant tasks). The task of the harness is sophisticated context management and giving the LLM access to tools.
Figure 1: An engine analogy for conventional LLMs, reasoning models, and agent harnesses. Figure from my Components of a Coding Agent article.
In this analogy depicted in the figure above, the model is the engine, and the agent harness adds the machinery that lets us use it for various things (as an upgrade from a simple chatbot interface). An agent can use either a conventional LLM or a reasoning model, but reasoning models are the more typical choice as they are more capable. (As mentioned above, most current-gen LLMs are reasoning models with variable reasoning effort settings.)
So, the bottom line is that reasoning models, as the “engine,” remain an important part of understanding modern LLM agents.
The resources accompanying these topics shown above are:
The Build a Large Language Model (From Scratch) book and videos
The Build A Reasoning Model (From Scratch) book (and videos below)
The Components of a Coding Agent article on Substack
The videos below focus on developing the reasoning model that can power such a system.
Scope and prerequisites
The reasoning-from-scratch videos focus on a small pre-trained Qwen3 model, where we implement the reasoning methods ourselves in Python and PyTorch. Starting with a pre-trained model lets us focus on the interesting part: evaluation, inference-time scaling, reinforcement learning, and distillation.
Regarding the prerequisites, I recommend being comfortable with Python and PyTorch. I designed the reasoning materials as standalone content, so the previous Build A Large Language Model (From Scratch) content is recommended but not strictly required.
Compute-resource-wise, you can work through the early text-generation and evaluation examples on a CPU. For the later experiments, especially training, a GPU makes the work much more practical. The first video covers the setup and some more recommendations; the repository also includes hardware and environment recommendations.
Figure 2: An overview of the topics being covered.
The companion book
These videos accompany my book, Build a Reasoning Model (From Scratch). If you like to read through the explanation in more detail and at your own pace, work through exercises, and so on, I recommend the book as a companion here.
For the most in-depth learning experience, my suggestion is to read a chapter, work through its code alongside the corresponding video, and then try the exercises.
The book is available from Manning and Amazon, and the print edition is in color!
Figure 3: A random excerpt from the Build a Reasoning Model (From Scratch) print book.
Thank you to everyone who has picked up a copy or recommended it to a friend. Your support helps me keep creating these materials.
The videos
As a last, quick side note before getting to the main videos below: For a shorter, more conceptual overview, I also recommend my AI Reasoning Models course on LinkedIn Learning (it should be free for 24h when accessing it via the link there, which shouldn’t then require a LinkedIn Premium subscription).
The course is about 90 minutes and explains how reasoning models work, covering the main ideas behind inference-time scaling, reinforcement learning, and distillation. It’s a good optional preview if you’d like the main ideas without working through the full implementations below.
To get to the main part of this article, here are the eight main lecture videos.
1. Motivation and code setup
The first video starts with an explanation of the relationship between conventional LLMs, reasoning models, and agents. Next, the main topic of the video is setting up the coding environment. This includes installing the Python and PyTorch dependencies (with uv), among others. I also discuss hardware choices and common setup issues.
Topics covered
The relationship between conventional LLMs, reasoning models, and agent harnesses
On the benefits of learning through code implementations
The scope of this video series
Installing Python dependencies
Choosing hardware and checking CUDA or Apple Silicon support
Using a remote GPU with VS Code and troubleshooting the environment
Watch video 1 · Setup instructions
2. Loading a base model, text generation, and KV caching
Before adding reasoning methods in the upcoming videos, we need a working base model. Instead of training it from scratch, since that would be a whole other book (see Build A Large Language Model (From Scratch)), we start with a pre-trained model.
Here, we load the pre-trained weights of a Qwen3 0.6B model. Then, we implement the text-generation process via a streaming text generator and add KV caching and model compilation to make text generation more efficient.
These implementations give us the baseline and base components we will evaluate and improve in the following videos.
Topics covered
Encoding text as tokens and decoding it back into text
Downloading Qwen3 weights and loading the model on the selected compute device
Going over tensor shapes, logits, and next-token predictions
Implementing greedy decoding and streaming text generation
Handling end-of-sequence tokens and measuring generation speed
Adding a KV cache to reuse attention keys and values
Combining KV caching with
torch.compileand comparing hardware performance
Watch video 2 · Chapter 2 code
3. Building a verifier for evaluation and reinforcement learning
In this lecture, we build a math-answer verifier step by step, from answer extraction to normalization and checking mathematical equivalence. We then use it to evaluate the model on MATH-500.
We use this verifier to evaluate the base model’s performance on the MATH-500 evaluation dataset.
However, the verifier also has a second job later in the series: providing the reward signal for reinforcement learning with verifiable rewards (RLVR).
Topics covered
Intro to verifiers and how they fit into LLM evaluation methods (other methods are covered in the book’s appendix)
Reusing answer verification as a reward signal for RLVR (more on that in chapter 6)
The verifier flow, including extracting final answers, normalizing answers and checking mathematical equivalence
Building an answer grader and an evaluation loop for MATH-500
Comparing base and reasoning models on MATH-500
Watch video 3 · Chapter 3 code
4. Inference-time scaling: sampling and self-consistency
Now we explore ways to improve answers while keeping the model weights fixed. But before implementing specific inference-scaling techniques, we first implement methods to generate diverse answers from the same prompt (this is standard practice in proprietary LLM chatbots, such as ChatGPT).
Concretely, we implement temperature scaling, multinomial sampling, and top-p filtering to generate diverse responses. Then we combine multiple responses through self-consistency, using majority voting over their final answers.
We also examine the tradeoff between answer accuracy and the extra computation needed to generate more candidates.
Topics covered
Comparing how additional compute benefits model performance during training and during inference
A gentle intro to inference-scaling through chain-of-thought prompts
Implementing temperature scaling and multinomial sampling to generate diverse answers
Adding top-p filtering to control the quality of the diverse answers
Combining candidate answers through self-consistency, one of the most popular inference scaling techniques
Evaluating MATH-500 accuracy against the cost of additional inference compute
Watch video 4 · Chapter 4 code
5. Inference-time scaling: scoring and self-refinement
Building on the previous video, we implement scoring functions to compare multiple answers generated from the same prompt. We can also use these scoring methods to break ties in self-consistency (covered in the previous lecture).
The scoring methods covered here include computing token and sequence log probabilities (this will also be relevant in the later reinforcement learning chapters). We then implement a self-refinement loop in which the model critiques an answer and attempts a revision.
The evaluation lets us check how useful those additional steps are for our model.
Topics covered
Using scoring functions for comparing multiple answers and breaking ties
Implementing a simple heuristic scorer
Computing token probabilities and sequence likelihoods (also relevant for the following lectures)
Using log probabilities for improved numerical stability
Building a self-refinement loop for generating critiques and revised answers
Evaluating self-refinement on MATH-500
Watch video 5 · Chapter 5 code
6. Reinforcement learning: implementing GRPO for RLVR
In this lecture, we move on to implementing a training technique called Reinforcement Learning with Verifiable Rewards (RLVR). We first implement a simplified version of Group Relative Policy Optimization (GRPO), generate groups of responses, score them with our verifier, and use their relative rewards for the model weight updates during training.
Topics covered
Generating reasoning traces and understanding the relationship between answer lengths and problem-solving abilities
Discussion and comparison of two important LLM post-training methods: Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)
Turning verifier scores into rewards and relative advantages
Computing sequence log probabilities and a simplified GRPO loss
Assembling the training loop with logging and checkpoint saves
Evaluating checkpoints and checking training stability and memory needs
Watch video 6 · Chapter 6 code
7. Reinforcement learning: tracking, clipped policy ratios, and KL terms
Now that we implemented the training loop in the previous lecture, we can revisit it and analyze the training runs in more detail.
Concretely, this video adds more training metrics for inspection and analysis, including entropy and advantage statistics, clipped policy ratios, KL loss term values, and formatted rewards.
We then review the training logs and evaluation results to understand what these additions do and whether additions such as clipped policy ratios and KL loss terms are helpful.
Topics covered
Running longer training experiments and interpreting their logs
Tracking advantages and entropy to understand training behavior
Implementing clipping in the policy loss and analyzing its effects
Adding a KL loss term and discussing reward hacking
Understanding limitations of the simplified KL formulation
Implementing format rewards and conditioning them on answer correctness
Comparing checkpoints on MATH-500 and exploring further GRPO research
Watch video 7 · Chapter 7 code
8. Distilling a reasoning model
Finally, we explore another way to train a small reasoning model, that is, learning from answers generated by a stronger (and bigger) teacher model. We discuss different forms of distillation, prepare reasoning data, and implement the supervised training loop.
We finish this lecture and course by evaluating the distilled model on MATH-500 and comparing the results. This rounds out the series with an alternative to developing reasoning capabilities through reinforcement learning.
Topics covered
Comparing the different combination and approaches to distillation, including soft and hard distillation, and on-policy and off-policy distillation
Creating and collecting teacher-generated answers for the MATH dataset (using DeepSeek R1)
Preparing the distillation dataset, including tokenization and training/validation/test splits
Computing answer-token log probabilities and understanding the cross-entropy loss
Implementing supervised training and tracking training and validation losses
Evaluating checkpoints on MATH-500 and comparing distillation results
Watch video 8 · Chapter 8 code
Final words
As you watch these lectures, I encourage you to pause occasionally and try a small experiment of your own if something interesting comes to mind.
For example, feel free to change the sampling settings, inspect a few failed answers, or try variations of different ideas similar to the suggestions in the exercises.
And if you would like the full written walkthrough, figures, and exercises alongside the videos, you can find Build a Reasoning Model (From Scratch) at Manning and Amazon.
Thanks for reading and supporting Ahead of AI. I am taking a short break to regenerate, and I look forward to being back with the next round of LLM research articles next month.
In the meantime, happy viewing and tinkering!
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 Sebastian Raschka(Building Reasoning Models from the Ground Up: A Full Video Course),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。



