11-19 更新 实时
首页/ 深度/ 正文

为什么一位计算机科学家认为超级智能不会出现

深度 来源:IEEE Spectrum AI 2026-11-19 20:00 0 阅读 机器翻译
为什么一位计算机科学家认为超级智能不会出现

关于 AI 超级智能的辩论存在严重缺陷 - IEEE Spectrum

为什么一位计算机科学家认为超级智能不会出现

面向技术内行

面向技术内行

IEEE Spectrum 是 IEEE 的旗舰出版物——IEEE 是全球最大的致力于工程与应用科学的专业组织。我们的文章、视频和信息图表让读者了解技术、工程和科学领域的最新进展。

创建账户,即可享受更多免费内容和权益

将文章保存下来以便日后阅读,需要一个 IEEE Spectrum 账户

The Institute 的内容仅会员可访问

下载完整 PDF 期刊仅限 IEEE 会员

下载本电子书仅限 IEEE 会员

访问 Spectrum 数字版仅限 IEEE 会员

关注主题是仅限 IEEE 会员的功能

加入全球最大的致力于工程与应用科学的专业组织,即可访问 Spectrum 的全部文章、档案、PDF 下载及其他权益。了解更多关于 IEEE 的信息 →

加入全球最大的致力于工程与应用科学的专业组织,即可获取本电子书以及 IEEE Spectrum 的全部文章、档案、PDF 下载及其他权益。了解更多关于 IEEE 的信息 →

访问数千篇文章——完全免费

创建账户即可获得专属内容和功能:保存文章、下载合集、发表评论——全部免费!如需完整访问权限和更多权益,请订阅 Spectrum。

为什么一位计算机科学家认为超级智能不会出现

但为此而努力仍可能带来灾难性后果

今年夏天,许多人措手不及:全球领先的 AI 公司之一的一些员工开始随口讨论,他们的产品是否会导致人类灭绝。事情始于 Anthropic 研究员 Jacob Coxon 宣布辞职,并警告说他的雇主和 OpenAI 正在“径直冲向可自我改进的超级智能,拿我们的生命冒险”。Anthropic 对齐科学负责人 Evan Hubinger 回应道:“Jacob 说得没错——我们确实真心相信 AI 可能杀死全人类!”随后,Anthropic CEO Dario Amodei 对 CNN 表示:“我对 Jacob 的认同远多于分歧。”

这些担忧源于一个在未来主义者圈子里流传了数十年的信念:一旦工程师设计出能够自我改进的 AI 系统,就会触发一个不断加速的自主升级循环,即所谓的递归自我改进(RSI)。其结果将是人工超级智能(ASI)——一种认知能力远超人类的系统,它将轻易摆脱我们的控制,甚至可能消灭我们这个物种。需要指出的是,人工超级智能是一个远比通用人工智能(AGI)更雄心勃勃的目标;AGI 通常指的是能在广泛的认知任务上达到人类水平的 AI。

这些观点引发了各不相同的反应。有人认为,ASI 这一概念正被用于营销,或者被用来推动有利于现有实验室打压新竞争对手的监管。另一些人则聚焦于这类系统是否真有可能把我们全部消灭,并围绕 AI 设计的超级病毒等具体情景的细节展开争论。还有人声称,让 ASI 与人类利益保持对齐是可行的,因此我们应当加速推进,迎来这个数字乌托邦。

在这些讨论中,有一个问题获得的关注相对较少:从工程学的角度看,超级智能究竟是否可能?这个答案很重要,因为它能帮助我们校准自身的焦虑,并针对这项新兴技术制定最有效的应对方案。

最近,我在华盛顿特区乔治城大学讲授的一门计算机科学博士研讨课上探讨了这一话题,我和学生们考察了超级智能叙事的技术与文化根源。(有兴趣的读者可以查看我们发布的课程阅读材料带注释书目。)我从中得出了若干针对 ASI 可行性的关键反对意见,下文将详细说明;同时我也产生了一种更深的担忧:即便所承诺的智能爆炸永远不会到来,追逐它本身仍可能造成实实在在的伤害。

反对意见 #1:递归自我改进只是推测

RSI 是一个老概念。在 1965 年的一篇论文中,统计学家 I.J. Good 推测,AI 自我改进的快速循环可能引发一场“智能爆炸”。这一想法在 1990 年代于湾区未来主义者团体 Extropians 中流行起来,因为它提供了一种讨论 ASI 的方式,而无需处理如何构建这样一个系统的工程细节。这一概念被 Eliezer Yudkowsky(后来创立了理性主义运动)和 Nick Bostrom(曾推动有效利他主义更多关注人类面临的生存风险)等知名人士所采纳。

然而最近,这个话题在 Anthropic(Claude 的开发者)和 OpenAI(ChatGPT 的开发者)内部都已从思想实验转向实际工程。在 6 月一篇题为“当 AI 构建自身”的博客文章中,Anthropic 描述了由 LLM 驱动的智能体如何加速代码的编写、调试和优化,并自动化了部分研究工作,例如运行实验和记录结果。该报告得出结论,公司的内部数据“正表明 AI 已经在加速 AI 系统的开发”。OpenAI 对其意图也同样明确;在 9 月的一份报告中,该公司宣布了到 2028 年 3 月开发出一名自动化研究员的目标。

但是,这些努力是否让这些公司走上了通向不可控 RSI 的轨道?有几个障碍削弱了这一说法。考虑一下改进 AI 系统所需数据的可获得性。截至 2023 年,大语言模型的重大进展都来自扩大模型规模。最初的 GPT-1 有 1.17 亿个参数(训练期间学到的值,塑造神经网络处理信息的方式)。这一数量在 GPT-2 上跃升至 15 亿,在 GPT-3 上达到 1750 亿,在三年前发布的 GPT-4 上(很可能)超过 1 万亿。然而,在 GPT-4 之后,继续扩大规模开始产生递减的回报。这促使工程师们越来越多地将注意力转向“后训练”阶段,在该阶段应用强化学习技术来提升已训练模型在特定任务上的表现。这种方法会奖励模型产生成功的响应,帮助它在未来更好地生成类似的答案。然而,它需要大量相关输入与正确输出配对的示例。

在计算机编程和数学这类高度结构化的领域,此类数据集相对容易生成,这有助于解释为什么我们在这些狭窄领域看到了迅速且令人瞩目的进展。在难以便捷获取这类训练数据的领域,进展则没那么显著。这或许能解释为什么 AI 能够攻克高深的定理,却难以清空一个电子邮件收件箱。“如果这条锯齿状前沿主要是数学+代码(借助[强化学习技术],你可以把它推进到任意远),”法国 AI 研究员 François Chollet 最近提出这样的假设,“而其他一切领域则开始趋于停滞,因为它仍然受制于人类生成的数据呢?”自动化编程智能体和由 LLM 驱动的科研人员的部署,并不会克服这一根本性限制。

声称 RSI 即将到来的说法,也低估了重大 AI 进展所需的大量科学工作。想想反向传播的关键作用——这种算法让超大型神经网络能够通过逐层调整内部参数来学习,以减少其预测中的误差。这一算法开启了深度学习革命,但它并非凭空出现在其创造者的头脑中。David Rumelhart、Geoffrey Hinton 和 Ronald Williams 在 1986 年发表的那篇颇具影响力的反向传播论文,建立在数学和计算机科学领域数十年关键突破的基础之上。

一场通向 ASI 的智能爆炸同样会依赖一长串重要性相当的创新,而这些创新又要建立在此刻尚未发生的诸多重大科学与工程进展之上。这将需要 AI 智能体以某种方式去模拟、组织并评估横跨众多领域、持续数十年的全球科研工作——这是一项雄心大到不可能完成的任务。事实上,由普林斯顿大学计算机科学家近期主导的研究发现,即便是高度受限的研究问题,当前的 AI 工具也难以自主取得进展。研究者总结道:“我们的结果提供了早期证据,表明如今的智能体能够完成 AI 研究中的工程部分,但在研究生命周期的关键环节上却举步维艰。”从当下这一现实,跃迁到同类工具正在将科学自动化的未来,其跨度之大几乎难以想象。

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Why a Computer Scientist Thinks Superintelligence Won’t Happen)

The AI Superintelligence Debate Has Serious Flaws - IEEE Spectrum

Why a Computer Scientist Thinks Superintelligence Won’t Happen

FOR THE TECHNOLOGY INSIDER

FOR THE TECHNOLOGY INSIDER

IEEE Spectrum is the flagship publication of the IEEE — the world’s largest professional organization devoted to engineering and applied sciences. Our articles, videos, and infographics inform our readers about developments in technology, engineering, and science.

Enjoy more free content and benefits by creating an account

Saving articles to read later requires an IEEE Spectrum account

The Institute content is only available for members

Downloading full PDF issues is exclusive for IEEE Members

Downloading this e-book is exclusive for IEEE Members

Access to Spectrum 's Digital Edition is exclusive for IEEE Members

Following topics is a feature exclusive for IEEE Members

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to this e-book plus all of IEEE Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →

Access Thousands of Articles — Completely Free

Create an account and get exclusive content and features: Save articles, download collections, and post comments — all free! For full access and benefits, subscribe to Spectrum .

Why a Computer Scientist Thinks Superintelligence Won’t Happen

Yet trying for it may still be disastrous

Pop-art robot head wired to colorful neural network nodes on patterned background

Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. It started when Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

These concerns stem from a belief that has circulated in futurist circles for many decades: Once engineers design an AI system capable of improving itself, it will trigger an accelerating cycle of autonomous upgrades known as recursive self-improvement (RSI). The result will be artificial superintelligence (ASI), a system with cognitive capabilities so vastly exceeding those of humans that it will easily evade our control and perhaps even wipe out our species. Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.

These ideas have prompted varied reactions. Some argue that the concept of ASI is being used for marketing or to instigate regulation that will help the incumbent labs squash new competitors. Others focus on the plausibility of such systems killing us all, debating the finer points of specific scenarios such as AI-engineered superviruses. Still others claim that aligning ASI to human interests is feasible and so we should rush ahead to usher in this digital utopia.

Amid these conversations, one question has received comparatively little attention: From an engineering perspective, Is superintelligence even possible? The answer is important because it can help us calibrate our anxieties and develop the most effective possible responses to this emerging technology.

I recently explored this topic in a computer science doctoral seminar that I taught at Georgetown University, in Washington, D.C., in which my students and I examined the technical and cultural roots of the superintelligence story. (For those who are interested, we published an annotated bibliography of our class readings.) I came away with several key objections to the feasibility of ASI, which I’ll detail below, as well as a deeper fear that even if the promised intelligence explosion never arrives, pursuing it can still create real harm.

Objection #1: Recursive Self-Improvement Is Speculative

RSI is an old concept. In a 1965 paper, statistician I.J. Good speculated that a rapid cycle of AI self-improvement could lead to an “intelligence explosion.” The idea gained traction in the 1990s among the Extropians, a Bay Area futurist group, where it offered a way to discuss ASI without having to address the engineering details of how one would build such a system. The concept was taken up by such notable figures as Eliezer Yudkowsky (who later founded the rationalism movement) and Nick Bostrom (who helped steer effective altruism to focus more on existential risks to humanity).

Recently, however, this topic has shifted from thought experiment to practical engineering within both Anthropic (makers of Claude) and OpenAI (makers of ChatGPT). In a June blog post titled “When AI Builds Itself,” Anthropic described how LLM-powered agents are speeding up the writing, debugging, and optimization of code, and also automating parts of research such as running experiments and logging results. The company’s internal data, the report concluded, “is showing that AI is already accelerating the development of AI systems.” OpenAI has been similarly explicit about its intentions; in a September report, the company announced a goal of developing an automated researcher by March 2028.

But do these efforts put the companies on a trajectory toward uncontrollable RSI? Several obstacles undermine this claim. Consider the availability of the data needed to improve AI systems. Through 2023, major advances in LLMs came from scaling up the size of the models. The original GPT-1 had 117 million parameters (values learned during training that shape how the neural network processes information). This quantity jumped to 1.5 billion for GPT-2, to 175 billion for GPT-3, to (likely) more than a trillion for GPT-4’s release three years ago. After GPT-4, however, continued scaling began yielding diminishing returns. This led engineers to increasingly turn their attention toward “post-training” phases, where techniques from reinforcement learning are applied to improve an already trained model’s performance on specific tasks. This approach rewards a model for producing successful responses, helping it better produce similar answers in the future. It requires, however, lots of examples of relevant inputs paired with correct outputs.

Such datasets are relatively easy to generate in highly structured fields such as computer programming and mathematics, which helps explain why we’ve seen rapid and impressive advances in these narrow domains. Progress in areas that lack easy access to this type of training data has been less dramatic. This might explain why AI can tackle advanced theorems but struggles to empty an email inbox. “What if the jagged frontier is mainly math + code (which you can push arbitrarily far with [reinforcement learning techniques]),” the French AI researcher François Chollet recently hypothesized, “and everything else starts to plateau because it is still bottlenecked by human generated data?” The deployment of automated coding agents and LLM-powered researchers won’t overcome this fundamental limitation.

Claims that RSI is imminent also understate the massive scientific efforts required for major AI advances. Consider the critical role of backpropagation, an algorithm that enables very large neural networks to learn by tuning internal parameters, layer by layer, to reduce errors in their predictions. This algorithm enabled the deep learning revolution, but it didn’t emerge ex nihilo in the minds of its creators. The influential 1986 paper on backpropagation by David Rumelhart, Geoffrey Hinton, and Ronald Williams built on decades of critical breakthroughs in mathematics and computer science.

An intelligence explosion leading to ASI would similarly depend on a long sequence of innovations of comparable importance, which would build on many significant science and engineering advances that haven’t yet happened. This would require AI agents to somehow simulate, organize, and evaluate decades of worldwide scientific effort across numerous fields—an impossibly ambitious task. Indeed, recent research led by computer scientists at Princeton found that current AI tools had difficulty making autonomous progress even on highly constrained research questions. “Our results provide early evidence that today’s agents can do the engineering of AI research,” the researchers conclude, “but struggle with critical parts of the research life cycle.” The leap from this current reality to a future in which these same types of tools are automating science is almost too massive to imagine.

Finally, the RSI narrative downplays resource limitations. Tens of thousands of human brains have worked for decades, aided by increasingly powerful computers, to develop today’s AI systems, which can match human intelligence in some domains. That represents a massive investment in computation of both the biological and digital variety. What if the next intelligence level requires double the resources, and then the next level requires double again? Soon there would be no number of chips or data centers that we could build to provide the necessary compute and keep advancing toward ASI.

Pulling together these threads, I believe that the idea of RSI leading to an intelligence explosion is highly speculative and based on many unproven assumptions. While RSI isn’t definitively impossible, the idea demands more technical skepticism than it’s currently receiving. Carl Sagan’s adage that extraordinary claims require extraordinary evidence certainly applies here.

Objection #2: Superintelligence Might Not Be Computable

There’s an implicit claim in the superintelligence discourse that computers are theoretically capable of any calculation that can be imagined, so long as you can find a sufficiently smart algorithm, running on sufficiently powerful digital processors. In this context, RSI is merely a process for rapidly uncovering the special algorithms that can implement unimaginable smarts.

We cannot simply assume that any science fiction scenario we conjure about powerful AI is possible if we give machines enough time to self-improve.

But this is not how computer scientists think about computation. In his canonical 1937paper, “On Computable Numbers,” Alan Turing proved a surprising and relevant result: Most problems cannot be solved by algorithms. His elegant proof established the space of possible problems to be uncountably infinite (like infinite precision decimal numbers), while the space of possible algorithms is only countably infinite (like whole numbers). Later breakthroughs in computational complexity theory established that of the problems that are technically solvable by computers, many would require an astronomically large amount of time and resources to crack, making them practically impossible to solve. Theoretical computer scientists, in other words, don’t assume that computers will solve everything; we’re instead excited whenever we find something that these machines happen to be able to efficiently handle.

We cannot simply assume, as the futurists often do, that any science fiction scenario we conjure about powerful AI is possible if we give machines enough time to self-improve. There might be some capabilities in which machines can obtain seemingly magical levels of performance, but there will also be many others that no machine, whether computer or human, will ever make much progress.

A relevant counterargument here emphasizes the speed of silicon chips compared to human neurons: Perhaps the sheer speed of digital computation can yield machines that run cognitive circles around our clunky brains, even if they’re not solving vastly more complicated problems than we’re already capable of tackling. But as the computer scientist and science fiction author Vernor Vingepointed out, rapidity in cognition is not the same as intelligence. Speeding up the mind of a dog, he argued, wouldn’t enable the animal to play chess. Implementing roughly human-level intelligence on a faster machine might be impressive, but it’s not necessarily alarming.

At the moment, there’s a growing list of tasks that AI systems can perform better than humans, including winning complex games, identifying software bugs, and proving certain types of math theorems. It’s tempting to extrapolate from this specific progress to general mastery of every ability you might imagine. But it’s important to remember that the frontier of AI capabilities has always been jagged, with some research directions yielding impressive results and many others proving to be dead ends—a reality that would not have surprised Turing. More likely than a future dominated by an all-powerful superintelligence system is one in which we continue to painstakingly explore the space of problems well suited to AI, creating useful tools but falling well short of the digital deity predicted by the futurists.

Objection #3: Alignment Isn’t Hard (Given the Right System Design)

Last fall, the prominent AI doomsday theorist Yudkowsky appeared on “The Ezra Klein Show” to talk about superintelligence. Theepisode was ominously titled “How Afraid of the A.I. Apocalypse Should We Be?” Klein and Yudkowsky quickly got into the difficulty of “alignment”—training an AI system to act in a manner that aligns with human interests. Yudkowsky argued that we are struggling to control our existing systems, which implies that we’ll have no chance of controlling the much smarter variations of this technology that are soon to follow.

To underscore his point, he referred to a disturbing incident described in thesystem card for OpenAI’s GPT-o1 model. OpenAI had set up a security challenge that instructed an LLM-powered coding system to break into a server to retrieve the contents of a protected file. But a configuration mistake prevented the server from launching, rendering the task impossible. Instead of giving up, the system exploited a security flaw to start a new instance of the server, adding a command that directly revealed the protected file’s contents. “It jumped out of the system,” explained Yudkowsky, adding: “This is not something that any human particularly programmed into it.”

Though this 2024 incident flew largely under the radar at the time, a more troubling episode made a bigger splash this summer. OpenAI revealed that one of the company’s models circumvented its restricted internet access to hack into Hugging Face servers, seeking information that would help it pass a cybersecurity evaluation—all without the knowledge of its human creators. OpenAI soon admitted that the Hugging Face attack was just one of many examples of its systems going rogue. An opinion article published in The Hill articulated a widely held reaction when it described this behavior as “something straight out of sci-fi.”

These examples seem to support Yudkowsky’s fear that AI will inevitably develop a mind of its own, a key step on the route to uncontrollable ASI. But a closer look reveals a less frightening picture. The systems involved in these hacking incidents were AI agents: programs that continually prompt an LLM to ask what actions to take next in pursuit of a goal. In the hacking incidents, this basic ask/act pattern was repeated in a long-running loop with minimal human supervision.

Using LLMs to suggest actions over long time horizons is a risky proposition. As any frequent chatbot user can attest, LLM outputs can be plausible and logical while still veering off in unexpected directions that violate commonsense human norms about what’s appropriate.

In the GPT-o1 incident, for example, it’s likely that the agent reported the error message caused by the server misconfiguration to the LLM. During its training, the LLM had probably encountered reports of similar error messages accompanied by instructions for how to restart the server in question, so it passed along similar instructions to the agent. This is a perfectly plausible response when considered in isolation, even if it led to actions that a human would immediately reject as violating the spirit of the experiment. Something similar helps explain the hacking attacks over the summer. The LLM suggesting a hack to succeed at a cybersecurity challenge is a perfectly plausible response, even if it’s not what the experimenters had in mind.

I believe that pursuing RSI is an act of clear negligence.

The misaligned behavior that troubles commentators like Yudkowsky, therefore, doesn’t necessarily reveal something fundamental about advanced AI. It’s instead an expected side effect of the particular strategy of letting LLM outputs determine the actions of an autonomous agent without sufficient constraints. An obvious way to address these risks is to consider alternative architectures. Fortunately, there are many examples of AI systems that can complete tasks at human or superhuman levels without relying solely on LLM output to determine their actions.

Consider Tesla’s rapidly improving autonomous driving systems, or DeepMind’s AlphaFold, which helped win its developers a Nobel Prize in Chemistry for its ability to predict protein structures, or the AI game engines that can now beat humans in many popular pastimes, including chess, poker, Go, and the strategy game Diplomacy. None of these systems, impressive as they are, create concerns about going rogue. Their construction prevents such an outcome: Instead of prompting an LLM to generate an open-ended strategy for pursuing a general goal, these AIs consider much more limited spaces of possible actions. AlphaFold never surprises its users because its architecture does only one thing: It maps amino acid sequences to 3D protein structures. Tesla’s cars never unexpectedly form vehicle swarms because they’re asked only to avoid immediate obstacles, not to craft convoluted plans for achieving long-term objectives.

The implication of recent AI hacking attacks, therefore, is not a reflection of the difficulties of alignment. It is instead evidence that autonomous systems powered primarily by LLM prompts are unpredictable and possibly unsafe. More task-specific architectures can produce impressive performance much more safely, if we’re willing to put in the effort to develop them.

The Dangers of Chasing Superintelligence

Even if I’m correct that superintelligence is not a realistic engineering goal, this doesn’t mean we’re safe from harm. Merely pursuing ASI, as both OpenAI and Anthropic seem intent on doing, can create problems. Why, for example, did OpenAI persist in running its problematic hacking experiments over the summer, even after multiple warnings that the agents were likely to act unpredictably? Perhaps the company judged these risks acceptable collateral damage in the race to be the first to bring ASI into the world.

This fascination with superintelligence might also explain why OpenAI and Anthropic have bothdescribed RSI as a critical goal. As argued earlier, RSI is a concept from futurism and not computer science, and it isn’t necessary to build useful AI products. But if you believe achieving ASI is your destiny, pursuing recursive self-improvement becomes a primary objective. This matters because even if RSI never produces an intelligence explosion, it will almost certainly lead to systems that are harder for humans to understand and control. Which is why I believe that pursuing RSI is an act of clear negligence.

What’s needed now is for engineers and scientists with no financial or ideological skin in the game to participate in the AI discourse. We cannot let a few AI companies with mixed motives dictate the terms for how we discuss this technology. If superintelligence is a shaky concept, experts need to say so. If agents powered by prompting LLMs aren’t the best path toward safe and productive AI, experts should argue that as well. For the past few years, we’ve let a small group of individuals from the Bay Area who have quirky beliefs and astonishing sums of money tell us how to think about their products. It’s time for those of us able to think for ourselves on these matters to stand up and declare, “We’ve heard what you had to say. Now it’s our turn.”

Related Articles Around the Web

{"imageShortcodeIds":[]}

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 IEEE Spectrum AI(Why a Computer Scientist Thinks Superintelligence Won’t Happen),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗