控制 LLM 的推理强度

距离 OpenAI 发布 o1 已经过去近两年,这款模型让基于 LLM 的推理模型这一理念广为人知。大约四个月后,DeepSeek-R1 问世,同时公开了用于训练此类推理模型的可验证奖励强化学习(RLVR)配方的细节。
上周,OpenAI 发布了 GPT-5.6 模型系列。该系列有三种规模,每种规模大约提供五到六档推理强度设置。
所以,推理模型确实会长期存在。它们已经成为现代模型发布中的标准组成部分。
过去,我介绍过推理模型的方法论(《理解推理 LLM》),也介绍过相关研究论文(《LLM 推理的强化学习现状》和《LLM 推理模型推理现状》)。我甚至还写了一本全新的 440 页的书,讲如何开发推理模型,名为《从零构建推理模型》(Build A Reasoning Model (From Scratch))。
这些资源聚焦于如何把常规 LLM 转变为推理模型。而现在,在本文中,我想重点解释如何开发一个具备多种推理强度模式的推理模型,类似于本文开头那幅图所示。
不用担心,本文可以独立阅读。不过,前面提到的那些资源可能也很有意思、很有用。
在谈论几乎任何机器学习或 AI 技术、子领域时,有一条经验是:我们通常不应该“字面地”理解技术术语。例如,机器学习和 AI 中的(人工)神经网络,其运作方式并不真的像人脑那样的生物神经网络。
同样,在谈到「推理模型」时,我们不应该指望这些模型真的像人类一样推理。在 AI 和 LLM 研究的语境中,「推理模型」指的是会输出中间推理轨迹的模型,这条轨迹就像是一个中间回复,一步步地推演某个问题或任务。
要解释这一点,最简单的办法大概就是看一个例子。
提升(推理)任务性能基本上有两种途径:训练扩展和推理扩展。
我们先简单谈谈训练。
简而言之,DeepSeek-R1 提出用可验证奖励的强化学习(RLVR)来训练 LLM,将其变成推理模型。RLVR 是一种为可验证的数据领域提供奖励信号(0=错误,1=正确)的技术。这里的可验证数据领域是数学(我们可以用 SymPy 或 WolframAlpha 这类符号数学检查器来验证结果)和代码(我们可以用编译器、单元测试,或 LeetCode 这类集成平台)来检查正确性。
值得注意的是,推理轨迹本身并没有被用于训练或更新模型。尽管他们尝试把这种中间回复信息用于训练,但 DeepSeek-R1 论文报告称它对模型训练没有帮助,所以最终没有被采用。(是否以及如何通过过程奖励模型把中间推理轨迹纳入训练信号,仍是一个活跃的研究领域。)
总之,如图 7 所示,仅靠输出奖励进行训练,就足以让模型学会如何推理一个问题,也就是说,它会学会写出中间解释、进行回溯并自我纠正。模型意识到自己犯了错误并自我纠正的这些时刻,被称为“Aha”时刻。
顺便说一句,虽然 DeepSeek-R1 无疑是更受关注的那篇论文,也是围绕可验证奖励的强化学习以及推理模型发展引发热潮的论文,但还有另一篇论文 Kimi K1.5,与它恰好在同一天(2025 年 1 月 22 日)发布于 arXiv。此外,RLVR 这一术语早在两个月前就已在 Tülu 3: Pushing Frontiers in Open Language Model Post-Training 中被提出。
DeepSeek R1 最终成为更受关注论文的一个原因,是它证明了推理行为可以通过纯强化学习(RL)实现。
例如,Tülu 3 和 Kimi K1.5 是在经过监督微调(SFT)的模型之上应用强化学习。DeepSeek-R1 模型同样是从 DeepSeek-V3 基础模型的 SFT 检查点开始训练的,其中还包含一个以纯 RLVR 训练的 DeepSeek-R1-Zero 变体。R1 Zero 是比 R1 更弱的模型,但它表明 RLVR 足以教会模型生成并使用推理轨迹。
虽然 R1-Zero 更像是一个概念验证模型,但请注意,完整的 DeepSeek-R1 推理模型训练流程通常是多阶段的,如上所述,也要更复杂一些。
顺便说一句,如今大多数 LLM 实际上都是推理模型,也就是说,它们是以类似 DeepSeek-R1 的方式、借助某种形式的 RLVR 训练出来的。
除了通过训练改进推理行为之外,提升模型性能的另一个杠杆是推理计算扩展。简而言之,这意味着我们在模型训练完成之后、使用过程中投入更多算力,以获得更好的答案。
这本身就是一个完整的主题,你可以阅读我的《The State of LLM Reasoning Model Inference》一文,获得更详细的梳理:
下面我会尽量总结作为背景信息最需要提及的内容。
首先,用 RLVR 训练模型本身就已经在隐式地带来某种形式的推理扩展,因为与传统 LLM 相比,推理模型在推理时通常会输出更多 Token,这意味着我们在推理期间投入了更多算力。
其次,我们可以通过推理努力等级(reasoning effort levels)进一步调整这一输出长度,不过这个稍后再详细说。
第三,还有许多额外的推理扩展技术。其中一种流行的方法是自洽性(self-consistency),它通常实现为一种多数投票的形式:对模型进行多次查询,最终答案通过多数投票选出。
该方法既可用于传统 LLM,也可用于推理模型。此外,这种方法可以按需使用,并且可以与推理训练叠加使用。一个很好的例子是 DeepSeekMath-V2,研究人员在一个(专攻数学的)推理模型之上应用了极端的推理扩展,从而在具有挑战性的数学奥林匹克类问题上达到最先进的性能。
不过,关于其他技术的概览,我还是要再次推荐我的另一篇文章《The State of LLM Reasoning Model Inference》:
你可能在前面“顿悟时刻”那张图里见过 <think></think> 这些 Token。我也把对应的图放在下面了,这样你就不用一路往上翻了。
就推理能力而言,这些 <think> 和 </think> 标签只是表面形式。它们并不会让模型去推理,也不是获得良好推理表现的必要条件。完全可以训练同一个模型而不带这些分隔符,很可能也能达到相近的基准测试表现。
这些 <think> 标签或 Token 的作用,主要是标记推理轨迹从哪里开始、到哪里结束,以便训练流程或用户界面能把它与最终答案分开,并可以选择对用户隐藏。(ChatGPT 或 Codex 这类界面通常会这么做。)
这里的要点是:<think> Token 并不会赋予模型“思考”或推理、或推理得更好的能力。完全可以训练同样的模型而不带这类 <think> Token,并达到相近的基准测试表现。
字面字符串 <think> 和 </think> 本身也没有什么特别之处。换另一对分隔符也能起到同样的作用。
顺便说一句,这种做法的实现方式通常是在 RLVR 阶段加入一项格式奖励。也就是说,除了根据答案正确性给模型奖励,还会对使用 <think> Token 额外给予奖励,从而鼓励模型去使用它们。
例如在 DeepSeek-R1 中,总奖励的计算方式为
R_total = R_accuracy + R_format
其中格式奖励是一个简单的基于规则的检查,鼓励模型把推理过程放在以下部分之内:
推理轨迹
第一代推理模型是专用推理模型。我的意思是,当时有一个 DeepSeek-V3 基础模型,以及一个独立的 DeepSeek-R1 推理模型。
无论提示词是什么,R1 通常都会输出非常冗长的回复,消耗大量 Token,即便是简单的提示词也不例外。它也没有内置选项可以关闭推理模式。
后来的模型,如 Qwen3 等,尝试了混合方式:同一个模型既可以像普通的指令微调模型那样运行,也可以按需表现为推理模型。
注:有些模型开发者称之为“思考模式”,另一些则称之为“推理模式”。这两个词指的是同一种行为。
在 Qwen3 中,这是通过分词器设置 enable_thinking=True 或 enable_thinking=False 来处理的。在底层,设置 enable_thinking=False 实际上会在助手回复的开头添加一个空的 <think></think> 段,从而关闭 Qwen3 的推理(“思考”)模式。
那么,在训练过程中是如何实现这一点的,使得模型在推理时能够支持这种开关,如上图所示?
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Controlling Reasoning Effort in LLMs)
It has been almost two years since OpenAI released o1, a model that popularized the idea of LLM-based reasoning models. DeepSeek-R1 followed about four months later, together with details of a reinforcement learning with verifiable rewards (RLVR) recipe to train such reasoning models.
Last week, OpenAI released the GPT-5.6 model family. It comes in three sizes, each with roughly five or six reasoning-effort settings.

So yes, reasoning models are here to stay. They have become a standard part of modern model releases.
In the past, I covered the methodology of reasoning models (Understanding Reasoning LLMs) as well as relevant research papers (The State of Reinforcement Learning for LLM Reasoning and The State of LLM Reasoning Model Inference). And I even wrote a whole new 440-page book on how to develop reasoning models, Build A Reasoning Model (From Scratch).

These resources have focused on turning a conventional LLM into a reasoning model. Now, in this article, I want to focus on and explain how to develop a reasoning model that has multiple effort modes, similar to what’s shown in the figure at the beginning of this article.
No worries, this article can be read as a standalone article. However, the aforementioned resources may be interesting and useful.
1. A brief definition of reasoning modelsWhen talking about pretty much any machine learning or AI technique or subfield, the one lesson is that we usually shouldn’t take technical terms “literally”. For example, an (artificial) neural network in machine learning and AI doesn’t literally work like a biological neural network like the human brain.
Similarly, when talking about “reasoning models”, we shouldn’t expect that these models literally reason like us humans. In the context of AI and LLM research, “reasoning model” means a model that outputs an intermediate reasoning trace, which is like an intermediate response that works through a question or task step by step.
It’s probably easiest to explain this by showing an example.

There are essentially two ways to improve (reasoning) task performance: training scaling and inference scaling.

Let’s briefly talk about training first.
2.1 Training reasoning models
In a nutshell, DeepSeek-R1 proposed training an LLM using reinforcement learning with verifiable rewards (RLVR) to turn it into a reasoning model. RLVR is a technique to provide a reward signal (0=incorrect and 1=correct) for verifiable data domains. These verifiable data domains here are math (we can use a symbolic math checker like SymPy or WolframAlpha to check results) and code (we can use a compiler or unit tests, or integrated platforms like LeetCode) to check for correctness.
Notably, the reasoning trace itself was not used for training or updating the model. Although they tried to use this intermediate response information for training, the DeepSeek-R1 paper reported that it wasn’t helpful for the model training, so it was ultimately not used. (Whether and how to incorporate intermediate reasoning traces in the training signal via process reward models is an active area of research.)

2.2 “Aha” moments
Anyway, just training on the output rewards alone, as Figure 7 shows, turned out to be sufficient for the model to learn how to reason through a problem, meaning that it would learn to write intermediate explanations, backtrack, and self-correct itself. These moments when the model realizes that it made a mistake and self-corrects itself are called “Aha” moments.

By the way, while DeepSeek-R1 is inarguably the more popular paper, and the paper that created excitement around reinforcement learning with verifiable rewards and the development of reasoning models, there is another paper, Kimi K1.5, published on exactly the same day on arXiv (22 Jan 2025). Also, the term RLVR was already coined two months earlier in Tülu 3: Pushing Frontiers in Open Language Model Post-Training.
One reason why the DeepSeek R1 is ultimately the more popular paper is that it demonstrated that reasoning behavior can be achieved with pure reinforcement learning (RL).

For instance, Tülu 3 and Kimi K1.5 applied reinforcement learning on top of a supervised fine-tuned (SFT) model. The DeepSeek-R1 model was also trained from an SFT checkpoint of the DeepSeek-V3 base model, and it included a DeepSeek-R1-Zero variant trained with pure RLVR. R1 Zero is a weaker model than R1, but it showed that RLVR is sufficient for teaching the model to generate and use reasoning traces.
While R1-Zero was more of a proof-of-concept model, note that the full DeepSeek-R1 reasoning model training pipeline is usually multi-stage and a bit more complicated, as mentioned above.

By the way, most of today’s LLMs are effectively reasoning models, meaning they have been trained in a similar fashion to DeepSeek-R1 using a form of RLVR.
2.3 Inference scaling in a nutshell
Next to improving reasoning behavior through training, another lever for improving model performance is inference compute scaling. In short, this means that we are spending more compute after training the model, during usage, to get better answers.
This is a whole topic by itself, and you could read through my The State of LLM Reasoning Model Inference for a more detailed rundown:
I will try to summarize what’s most essential to mention as background info below.
First, training a model with RLVR is already implicitly leading to a form of inference scaling, since reasoning models usually output more tokens during inference compared to conventional LLMs, and that means we are spending more compute during inference.
Second, we can further adjust this output length via reasoning effort levels, but more on that later.
Third, there are many additional inference scaling techniques. A popular one is self-consistency, which is often implemented as a form of majority voting where the model is queried multiple times, and the final answer is selected via majority vote.
This can be applied to conventional LLMs as well as reasoning models. Also, this method can be used on demand and in addition to reasoning training. A good example of that is DeepSeekMath-V2, where the researchers applied extreme inference-scaling on top of a reasoning model (specialized for math) to achieve state-of-the-art performance on challenging math olympiad-type problems.

But again, I will refer to my other article, The State of LLM Reasoning Model Inference for an overview of other techniques:
3. Think tokensYou may have seen the <think></think> tokens in the earlier “Aha moments” figure. I also included the corresponding figure below so you don’t have to scroll all the way up.
These <think> and </think> tags are cosmetic with respect to reasoning ability. They do not make the model reason, and they are not required to achieve good reasoning performance. One could train the same model without these delimiters and likely reach similar benchmark performance.
The purpose of these <think> tags or tokens is mainly to mark where the reasoning trace begins and ends so that the training pipeline or user interface can separate it from the final answer and optionally hide it from the user. (UIs like ChatGPT or Codex usually do this.)
The point here is that the <think> tokens are not giving the model the ability to “think” or reason or reason better. One could train the same models without such <think> tokens and reach similar benchmark performance.
There is also nothing special about the literal strings <think> and </think>. Another pair of delimiters could serve the same purpose.
By the way, the way this is implemented is typically by adding a formatting reward during the RLVR stage. So instead of just rewarding the model based on answer correctness, one would provide additional reward for the use of <think> tokens, which in turn encourages the model to use those.
In DeepSeek-R1, for example, the overall reward was calculated as
R_total = R_accuracy + R_format
where the format reward was a simple rule-based check that encouraged the model to place its reasoning inside:
<think>
reasoning trace
</think>.
The first generation of reasoning models was dedicated reasoning models. With that, I mean that there was a DeepSeek-V3 base model and a separate DeepSeek-R1 reasoning model.
No matter what the prompt is, R1 generally outputs very verbose responses using lots of tokens, even for simple prompts. It also lacks a built-in option to turn off the reasoning mode.
Later models, like Qwen3 and others, experimented with hybrid approaches, where the same model can behave like a regular instruction fine-tuned model or a reasoning model on demand.
Note: Some model developers call this “thinking mode,” while others call it “reasoning mode.” Both terms refer to the same behavior.
In Qwen3, this is handled via the tokenizer using enable_thinking=True or enable_thinking=False. Under the hood, setting enable_thinking=False essentially adds an empty <think></think> section to the beginning of the assistant response to turn off Qwen3’s reasoning (”thinking”) mode.

thinking=False and thinking=True. (The empty <think></think> tags are hidden in the interface on the left as they are part of the modified input prompt, not the generated answer.)How is this implemented during training, such that the model supports this toggle during inference time, as shown in the figure above?
In short, as explained in the Qwen3 technical report, this on/off behavior is introduced primarily through supervised fine-tuning (SFT) and then reinforced during general RL in their largest flagship models.
For instance, after the initial reasoning model is trained via long-chain-of-thought SFT and reasoning RL, they add a “Thinking Mode Fusion” stage. During this additional SFT stage, the model sees both thinking and non-thinking examples:
/think: <think>{reasoning}</think>{answer}/no_think: <think></think>{answer}
Thinking is the default behavior, so /think can also be omitted. The subsequent general RL stage further reinforces this mode and format following.
These /think and /no_think flags are a “soft” switch. However, the enable_thinking=False setting mentioned earlier, which force-adds the empty <think></think> in the False case, acts then as a “hard” switch.

In other words, the tokenizer does not add /no_think to the query. It directly fills in the empty <think></think> section at the beginning of the assistant response. The model only sees the resulting tokens and continues directly with the answer.
Anyway, this on-and-off toggle is essentially a simplified version of the reasoning effort levels in GPT-5.6 and others, which I cover in the next section.
5. How “reasoning effort” settings workIn this section, I want to provide a brief overview of how the different reasoning effort toggles may be implemented, which have been introduced in models like GPT 5 and are present in pretty much any flagship model today.
Concretely, at the beginning of this article, I showed a figure from the Codex GPT 5.6 interface that lets users select multiple reasoning “effort” settings.
The following subsection will illustrate how these settings may be implemented. Then, in the next section, I will go over some of the more interesting research papers related to this topic.
5.1 Reasoning effort and response length and quality
Unfortunately, the implementation details of their effort settings are not shared by OpenAI, but there is some evidence out there that can be used for educated guesses.
For instance, via their open-source gpt-oss models from last year (I wrote about them in From GPT-2 to gpt-oss: Analyzing the Architectural Advances), we know that OpenAI allows us to toggle the reasoning effort setting via the system prompt (”Reasoning effort: low/medium/high”) that is prepended to each prompt.

As expected, the reasoning effort directly affects the response length and accuracy, as shown below.

Presumably, their GPT 5 models, including the recent GPT 5.6 models, use a similar approach.
By the way, note how different effort settings scale the response length in the figure above. The effort level seems directly correlated to token usage, which in turn seems correlated to accuracy. It might be possible to come up with effort settings beyond the “high” one, but I assume performance would saturate at some point. This saturation can be seen more clearly for the GPT 5.6 Sol model, which also shows that increasing reasoning budgets can become uneconomical at some point.
原文发布于 Sebastian Raschka(Controlling Reasoning Effort in LLMs),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。




