11-19 更新 实时
首页/ 算力/ 正文

AI 监管可向全球最危险行业学什么

算力 来源:TechXplore 2026-11-18 02:00 0 阅读 机器翻译
AI 监管可向全球最危险行业学什么

继续使用 Google、Facebook 或 Apple,即表示你同意使用条款。

AI 监管可以从全球最危险的行业中学到什么

作者:Palie Smart,The Conversation

编辑:Lisa Lock,审阅:Andrew Zinin

本文已按照 Science X 的编辑流程与政策进行审阅。编辑在确保内容可信度的同时,突出了以下属性:

由研究人员撰写

添加到首选来源

议员 Darren Jones 曾在 Keir Starmer 政府中担任首相首席秘书,他最近呼吁制定一项新的跨国条约,以规范超级智能(即 AI 在大多数认知任务上大幅超越人类)的安全开发与使用。

他还提出,英国在AI安全与可信度方面日益增长的声誉,使其成为开展此类合作的良好渠道。

这次发言中最令我感兴趣的,并不是AI是否可能对人类构成威胁这一问题。真正让我关注的,是潜藏其下的一个管理与组织问题:“我们的治理体系能否在确保AI安全的同时,跟上AI智能的发展步伐?”

我们正处在一个历史时刻:全球科技领袖及其部分员工认为,AI在未来十年内杀死全人类的概率超过10%。但人类至今仍难以理解AI可能带来的后果;等到我们真正明白时,或许已经来不及采取行动。

然而,我们必须谨慎地将这一预测与当前的风险区分开来,也要与AI系统能够独立自我改进(完全递归自我改进)并迈向超级智能时可能发生的情况区分开来。

我赞同呼吁缔结多国条约、并由英国发挥主导作用的主张。作为一名商业学者,我也相信,一些商学院的研究能够帮助我们理解如何管理和组织那些复杂性、速度和自主性正以超过我们理解能力的速度增长的系统。

这在核电和航空等高危行业中是熟悉的领域。在这里,有两种理论尤为相关:与美国社会学家Charles Perrow相关的常态事故理论,以及与美国政治学家Todd LaPorte的研究相关的高可靠性理论。将二者结合起来,可以为思考AI安全与监管提供一个有力的框架。

正常事故理论始于一个令人不安的命题。在复杂且相互依赖的系统中——例如核电站或为载人而建造的航天器——事故未必是个人无能、疏忽或设计糟糕的结果。它们可能以难以预料的互动方式,从多个组件的相互作用中涌现出来。

在这类系统中,事件发生得很快。当意外情况出现时,从一个潜在故障到下一个故障之间,可能几乎没有时间介入。这可能引发连锁错误,并有可能酿成灾难。一个常被引用的例子是 1978 年美国宾夕法尼亚州三里岛核电站的部分堆芯熔毁。这起事故是机械故障、仪表失效和人为因素共同作用的结果。

当你把 AI 智能体加入这样一个复杂系统时,风险会进一步上升——AI 智能体是指按照指令执行任务的系统。例子可能包括金融系统、软件开发系统、实验室中的科学数据库,甚至连接大量智能体的系统。

在任何此类系统中,单个组件——包括 AI 智能体——都可能达到可接受的质量阈值;难处在于它们之间的相互作用,以及我们对这些相互作用的理解。部分问题在于,我们知道智能体的行为可能不可预测:近期的一些报告描述了 AI 智能体以开发者未曾预料到的方式行事。

英国政府已强调了与能力日益增强的 AI 智能体相关的风险,并强调了对前沿 AI 模型进行测试、以及对其能力形成科学理解的重要性。

如果 AI 系统变得更加自主、更加互联、运行速度更快,那么依靠「人在回路中」来纠正重大错误的传统思路,可能会变得越来越难以实现。

除了我们自己造就的命运,别无命运?

高可靠性理论提供了一个对立视角。高可靠性这一传统源于对某些组织的研究——这些组织所运营的技术一旦失效可能带来灾难性后果,例如核电站、航空母舰和空中交通管制。

其核心洞见几乎与对正常事故理论的悲观解读截然相反。本质上,它认为组织可以通过形成特定的实践与文化,以极低的故障率运营极其复杂的技术。事实上,我们在电力系统、太空旅行和生物工程中已经看到了这一点。

这些组织之所以能实现高可靠性,是因为它们的基线假设就是自己并不安全。它们对更大问题的蛛丝马迹保持警觉,不会自动忽视异常现象,从险情和失败中学习,并构建冗余——例如为离线系统做备份并提供紧急关停开关。

它们常态化地演练故障,建立运营流程,为安全与保障而遵守监管,遵循专业标准和认证体系,并接受独立监督。高可靠性组织赋予专家挑战决策的权力。它们不断从自身表现中学习。

这为思考 AI 治理提供了一种截然不同的方式。我们不应只问「我们该如何监管 AI?」,而应问:「如果 AI 被嵌入到一个组织的方方面面,一个高可靠性组织会是什么样子?」

这表明,AI 安全不能简化为模型在部署前通过一次性的技术测试。它必须跨越不同组织,并超越国界。

这正是当前关于 AI 监管的政治讨论可能过于狭窄的地方。政府需要了解前沿实验室正在开发什么。但政府不可能现实地长期驻扎在每一家 AI 公司内部,检查每一个模型。

立法也无法预见到 AI 将呈现的每一种技术形态。英国政府认识到,必须通过保障(assurance)来培育 AI 的可信度。保障要问的是:是否有可信证据表明系统在既定边界内运行,这些边界是否恰当,风险是否受到监测,以及当边界被突破时是否有人负责。

航空业已有工程标准、认证制度、专业资质、独立检查、事故报告和监管监督。

核电在一个精密复杂的安全工程与监管保障生态系统中运行。金融机构和医药行业都拥有成熟的保障体系。

AI 正变得越来越具有重大影响,但其保障基础设施相对而言仍不成熟。考虑到前沿系统的发展速度,这一点尤其引人注目。制度层面的挑战不仅仅是出台更多 AI 监管,而是建立一种专业的安全文化,使监管具有实际意义。

一项关于安全、受监管地开发超级智能的潜在新多国条约,将把 AI 安全界定为一个快速演变的全球性挑战。与此同时,它可能为英国带来一个千载难逢的机会,帮助塑造正在成形的 AI 治理架构。

由 The Conversation 提供

这篇报道的背后是谁?

艺术史学士,物质文化硕士。曾任博物馆编辑、急救员和器官移植协调员。自 2021 年起为 Science X 担任编辑。完整简介 →

物理学硕士,有科研经历。资深科学新闻爱好者。在 Science X 的编辑工作中发挥关键作用。完整简介 →

本文依据 Creative Commons 许可转载自 The Conversation。阅读原文。

引用:AI 监管能从世界上最危险的行业中学到什么(2026 年 11 月 17 日),2026 年 11 月 17 日检索自 https://techxplore.com/news/2026-11-ai-hazardous-industries-world.html

本文档受版权保护。除为私人学习或研究目的进行的任何合理使用外,未经书面许可,不得复制任何部分。内容仅供参考。

在AI Safety Summit召开前夕,顶尖专家表示,世界各国领导人仍需正视AI带来的风险

监管AI看似是一项不可能完成的任务,但从伦理和经济层面来看,这却至关重要

在世界各国竞相争夺优势之际,AI的发展需要得到妥善监管

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(What AI regulation could learn from the most hazardous industries in the world)

By continuing with Google, Facebook or Apple, you agree to the Terms of Use.

What AI regulation could learn from the most hazardous industries in the world

by Palie Smart, The Conversation

edited by Lisa Lock, reviewed by Andrew Zinin

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

written by researcher(s)

Add to Preferred Sources

AI protest Credit: Unsplash/CC0 Public Domain

Darren Jones MP, the former chief secretary to the prime minister under Keir Starmer, recently called for a new multinational treaty governing the safe development and use of superintelligence (where AI substantially surpasses humans across most cognitive tasks).

He also argued that Britain's growing reputation for AI safety and trustworthiness made it a good conduit for such collaboration.

What I found most interesting about this intervention was not the question of whether AI could pose a threat to humanity. Instead, it was a management and organizational question sitting underneath it: "Can our systems of governance keep pace with the development of AI intelligence while also ensuring it is safe?"

We are at a moment in history where global tech leaders and some of their employees believe there is a greater than 10% chance that AI could kill all humans within the next decade. But humans are still struggling to understand the potential consequences of AI; by the time we do, it may be too late to act.

However, we must be careful to distinguish this prediction from the current risks and what might happen as AI systems become able to independently improve themselves (full recursive self-improvement) and move toward superintelligence.

I concur with calls for a multinational treaty and for the U.K. to take a leading role. As a business scholar, I also believe research from some business schools can inform us about managing and organizing systems whose complexity, speed and autonomy are increasing faster than our capacity to understand them.

This is familiar territory in high-hazard industries such as nuclear energy and aviation. Here, two theories are particularly relevant: normal accident theory, associated with the U.S. sociologist Charles Perrow, and high reliability theory, associated with the work of American political scientist Todd LaPorte. Taken together, they could provide a powerful framework for thinking about AI safety and regulation.

Normal accident theory begins with an uncomfortable proposition. In complex, interdependent systems, such as nuclear power plants or spacecraft built to carry humans, accidents are not necessarily the consequence of individual incompetence, negligence or bad design. They can emerge from the interaction of multiple components in ways that are difficult to anticipate.

In such systems, events happen rapidly. When something unexpected takes place, there may be little time to intervene between one potential failure and the next. This can lead to a cascading error, with the potential to cause a catastrophe. An often-used example is the partial meltdown at the Three Mile Island nuclear plant in Pennsylvania, U.S., in 1978. The accident occurred because of a combination of mechanical malfunction, instrument faults and human factors.

The risks further increase when you add AI agents into such a complex system—AI agents are systems that carry out tasks in response to instructions. Examples might include financial systems, software development systems, scientific databases in labs or even systems that connect lots of agents.

In any such system, an individual component—including the AI agent—might meet acceptable quality thresholds; the difficulty lies in the interactions between them and our understanding of these interactions. Part of the problem is that we know agents' behavior can be unpredictable: recent reports have described AI agents behaving in ways that their developers did not expect.

The U.K. government has highlighted risks associated with increasingly capable AI agents and has stressed the importance of testing frontier AI models and developing a scientific understanding of their capabilities.

If AI systems become more autonomous, more interconnected and faster moving, the conventional idea of a "human in the loop," to correct major errors, could become increasingly challenging.

No fate but what we make?

High reliability theory offers a counterpoint. The high-reliability tradition emerged from research into organizations that were operating technologies where failure could have catastrophic consequences, such as nuclear power plants, aircraft carriers and air traffic control.

Its central insight is almost the opposite of a pessimistic reading of normal accident theory. Essentially, it says that organizations can develop practices and cultures that allow them to operate extraordinarily complex technologies with very low failure rates. In fact, we have seen this with the electricity system, space travel and bioengineering.

These organizations achieve high reliability because their baseline assumption is that they are not safe. They remain attentive to hints of bigger problems, do not automatically dismiss anomalies, learn from near misses and failures and build redundancy—for example, backing up offline systems and providing kill switches.

They routinely rehearse failure, establish operational processes, adhere to regulation for safety and assurance, comply with professional standards and certification schemes and have independent oversight. High-reliability organizations give experts the authority to challenge decisions. They continuously learn from their performance.

This provides a very different way of thinking about AI governance. Instead of asking simply, "How do we regulate AI?" we should ask: "What would a high-reliability organization look like if AI was embedded throughout it?"

This suggests that AI safety cannot be reduced to a model passing a one-off technical test before deployment. It must span different organizations and transcend international boundaries.

This is where the current political conversation about AI regulation risks being too narrow. Governments need to understand what frontier laboratories are developing. But government cannot realistically sit permanently inside every AI company and inspect every model.

Nor can legislation anticipate every technical configuration that AI will take. The U.K. government recognizes it must cultivate trustworthiness in AI through assurance. Assurance asks whether there is credible evidence that a system is behaving within defined boundaries, whether those boundaries are appropriate, whether risks are being monitored and whether someone is accountable when they are breached.

The aircraft industry has engineering standards, certification regimes, professional qualifications, independent inspection, incident reporting and regulatory oversight.

Nuclear power operates within an elaborate ecosystem of safety engineering and regulatory assurance. Financial institutions and medicine both have sophisticated systems of assurance.

AI is becoming increasingly consequential, yet its assurance infrastructure remains comparatively immature. This is particularly striking given the speed at which frontier systems are developing. The institutional challenge is not simply to produce more AI regulation, but to create the professional safety culture that makes regulation meaningful.

A potential new multinational treaty on the safe and regulated development of superintelligence would frame AI safety as a rapidly evolving global challenge. At the same time, it potentially creates a once-in-a-generation opportunity for the U.K. to help shape the emerging architecture of AI governance.

Provided by The Conversation

Who's behind this story?

Lisa Lock

BA art history, MA material culture. Former museum editor, paramedic, and transplant coordinator. Editing for Science X since 2021. Full profile →

Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile →

This article is republished from The Conversation under a Creative Commons license. Read the original article.The Conversation

Citation: What AI regulation could learn from the most hazardous industries in the world (2026, November 17) retrieved 17 November 2026 from https://techxplore.com/news/2026-11-ai-hazardous-industries-world.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

World leaders still need to wake up to AI risks, say leading experts ahead of AI Safety Summit

Regulating AI seems like an impossible task, but ethically and economically, it's a vital one

Developments in AI need to be properly regulated as the world scrambles for advantage

The world is finally starting to regulate artificial intelligence—what to expect from US, EU and China's new laws

Calls to regulate AI are growing louder, but how exactly do you regulate a technology like this?

Video and 3D models document reclaimed materials' flaws to help builders plan reuse

Brain scans may explain why deepfake faces appear 'hyperreal'

Transformer shortages loom for strained US power grid

Neural networks that behave identically reveal hidden structural differences as they learn

Chat-based AI lets users run complex alloy calculations on their own computers

Popular messaging apps offer less anonymity than you think, research finds

Get Instant Summarized Text (Gist)

Complex, interconnected AI systems can produce cascading failures even when individual components meet quality standards. Safety governance should extend beyond one-off model tests to continuous monitoring, incident reporting, redundancy, independent oversight and international coordination, drawing on practices in high-hazard industries.

This summary was automatically generated using LLM. Full disclaimer

This site uses cookies to assist with navigation, analyse your use of our services, collect data for ads personalisation and provide content from third parties. By using our site, you acknowledge that you have read and understand our Privacy Policy and Terms of Use.

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 TechXplore(What AI regulation could learn from the most hazardous industries in the world),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗