苦涩教训提出者给 LLM 规模扩张派的新苦涩教训

由 THE DECODER 使用 Nano Banana Pro 生成
图灵奖得主 Richard Sutton 是 2019 年那篇影响深远的文章《苦涩的教训》的作者。他在一篇新文章中写道,仅靠原始算力并不足以实现智能。AI 科学需要巧妙的新方法,而且“远未完成”。
Sutton 此前已经批评过单纯扩展大语言模型的局限,称合成数据是一个“重大错误”,并呼吁打造具备世界模型、能持续从自身经验中学习的智能体。
Sutton 认为,真实世界无限复杂,而人类专业知识构成瓶颈,阻碍了真正的扩展。
《苦涩的教训》的提出者,给 LLM 的拥趸们带来了一堂苦涩的教训。其他研究者也表示认同。
在最近的一篇文章中,图灵奖得主 Richard Sutton 表示,仅靠原始算力并不足以实现智能。AI 科学需要巧妙的新方法,而且“远未完成”。他的文章在 X 上的 AI 社区引发广泛讨论,这说明 Sutton 触动了人们的神经。
在 2019 年那篇著名的《苦涩的教训》中,Sutton 提出,随算力扩展的通用方法最终会胜过精巧的人工设计方法。真正重要的是那些能随算力增长、而非依赖人类假设的思路。
Sutton 的立场并不新鲜。在早前的表态中,他就批评过单纯扩展大语言模型的局限。他称合成数据是一个“重大错误”,因为真实世界无限复杂,任何对其的模拟都只是“微观的”。人类专业知识构成瓶颈,阻碍了真正的扩展。
相反,他呼吁打造具备世界模型、能从自身经验中持续学习的智能体。世界远比互联网文本所能捕捉的复杂得多,Sutton 认为,如今的语言模型在训练结束后就不再学习。真正需要的是在不破坏已有知识的前提下持续学习。
AI 研究者 Jerry Tworek 与许多人一样,离开了主要 AI 公司之一的 OpenAI,去探索新的 AI 路径,他把 Sutton 的最新表态放在更大的背景中来看。Tworek 写道,“苦涩的教训”依然成立,但投入模型训练的计算越多,算法就越重要。
有些算法能够扩展,另一些则会走进死胡同,而扩展的许多维度仍完全未被探索。Transformer 扩展带来的暂时成功,加上对“苦涩的教训”的误读,“已经‘奖励黑客’了整整一代科学家的大脑。”
AI 研究者 Jayden Teoh 补充了另一个角度。他说,他多次看到“苦涩的教训”拖累学术研究,一些有前景的想法因偏离这一理念太远而被否定。Teoh 认为,这不是 Sutton 的错,而是对其原意的误读。
Teoh 还认为,事后看来似乎并不遵循“苦涩的教训”的想法,往往会被更简洁、更通用的方法取代。他说,这并不意味着那些想法是错的;它们往往正是当时推动进步的原因。他希望研究界不要一出现这类工作就将其否定。“一个想法并不一定要永远经受住扩展的考验,才能算作对 AI 进步的重要贡献,”Teoh 写道。
Sutton 呼吁新的学习方法,Tworek 则批评盲目扩大 transformer 规模,两者之间的张力正体现在一波不断增长的初创公司身上。在最新一期 Frontier Radar #5 中,我们追踪了 45 家正是押注这些空白地带的新 AI 实验室。
Tworek 本人经营着 Core Automation,正是这些实验室之一,致力于寻找超越预训练和强化学习的学习算法。这些创始人的共同点——包括创办 Oak Lab 的 Sutton——在于他们相信 AI 的能力会远胜今日,但怀疑仅靠当下这套配方无法实现。
一些人仍押注 LLM 规模扩展
知名行业领袖的看法则截然不同。OpenAI CEO Sam Altman 认为大语言模型的规模扩展远未走到尽头,并指责 LLM 批评者过于笃定规模扩展做不到什么,从而拖慢了一整代研究者。
作为证据,他提到 OpenAI 的一个模型最近推翻了一个数学家长期未能破解的数学猜想。他认为世界模型有用,比如在机器人领域,但表示数据明显支持继续扩大 LLM 规模。
Anthropic CEO Dario Amodei 持类似观点,称规模扩展的进展是一道没有尽头的彩虹。作为两家整个技术路线都押注于此的 AI 公司的 CEO,Amodei 和 Altman 显然在这一押注上有偏向。
没有炒作的 AI 新闻——由人工精选
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts)
Nano Banana Pro prompted by THE DECODER
Turing Award winner Richard Sutton, author of the influential 2019 essay "The Bitter Lesson," writes in a new post that raw compute alone isn't enough for intelligence. AI science needs clever new methods and is "far from finished."
Sutton had already criticized the limits of simply scaling large language models, called synthetic data a "big mistake," and called for agents with world models that continuously learn from their own experience.
The real world is infinitely complex, Sutton argues, and human expertise as a bottleneck prevents true scaling.
The inventor of the bitter lesson has a bitter lesson for LLM enthusiasts. Other researchers agree.
In a recent post, Turing Award winner Richard Sutton says raw compute alone isn't enough for intelligence. AI science needs clever new methods and is "far from finished." His post sparked wide discussion in the AI community on X, a sign that Sutton struck a nerve.
In his famous 2019 essay "The Bitter Lesson," Sutton argued that general methods scaling with compute beat sophisticated human-engineered approaches eventually. What matters are ideas that grow with compute rather than rely on human assumptions.
Sutton's position isn't new. In earlier statements, he criticized the limits of simply scaling large language models. He called synthetic data a "big mistake" because the real world is infinitely complex and any simulation of it is only "microscopic." Human expertise as a bottleneck prevents true scaling.
Instead, he called foragents with world models that continuously learn from their own experience. The world is massively more complex than what internet text can capture, and today's language models don't keep learning after training, Sutton argues. What's needed is continuous learning without destroying old knowledge.
AI researcher Jerry Tworek, who like many others left OpenAI, one of the major AI companies, to explore new AI approaches, puts Sutton's latest statement in context. The bitter lesson still holds, but the more compute goes into model training, the more the algorithm matters, Tworek writes.
Some algorithms scale, others hit dead ends, and many dimensions of scaling remain completely unexplored. The temporary success of transformer scaling, combined with a misreading of the bitter lesson, "has reward-hacked the brains of a whole generation of scientists."
AI researcher Jayden Teoh adds another angle. He says he has repeatedly seen the bitter lesson hold back academic research, with promising ideas dismissed for straying too far from its spirit. Teoh argues that this is not Sutton’s fault but a misreading of his original message.
Teoh also argues that ideas that seemed, in hindsight, not to follow the bitter lesson were often replaced by cleaner, more general approaches. That did not make those ideas wrong, he says; they were often what enabled progress at the time. He wants the research community to stop dismissing such work as soon as it appears. “An idea doesn’t necessarily have to survive scaling forever to be an important contribution to AI progress,” Teoh writes.
The tension between Sutton's call for new learning methods and Tworek's criticism of blind transformer scaling shows up in a growing wave of startups. In the latest Frontier Radar #5, we tracked 45 new AI labs betting on exactly these gaps.
Tworek himself runs Core Automation, one of those labs, searching for learning algorithms beyond pretraining and reinforcement learning. What unites these founders, including Sutton with Oak Lab, is that they believe in much more capable AI but doubt today's recipe alone will get there.
Some still bet on LLM scaling
Prominent industry leaders see things very differently. OpenAI CEO Sam Altman considers large language model scaling far from exhausted and accuses LLM critics of slowing down an entire generation of researchers by being too sure about what scaling can't do.
As evidence, he points to an OpenAI model that recently disproved a mathematical conjecture mathematicians had long failed to crack. He considers world models useful, for instance in robotics, but says the data clearly favor further LLM scaling.
Anthropic CEO Dario Amodei argues similarly, calling scaling progress a rainbow with no end. As CEOs of the two AI companies whose entire technical approach rests on this bet, Amodei and Altman are obviously biased in its favor.
AI News Without the Hype – Curated by Humans
原文为英文,正文由自动抓取并保留原文。
原文发布于 The Decoder(The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。