11-19 更新 实时
首页/ 深度/ 正文

机器人的安全功能已就位,但输入若说谎呢?

深度 来源:IEEE Spectrum AI 2026-10-28 00:14 0 阅读 机器翻译
机器人的安全功能已就位,但输入若说谎呢?

深入解析 AI 机器人的功能安全与网络安全 - IEEE Spectrum

IEEE Spectrum 11 月刊来了!

你的机器人的安全功能本身运转正常。可如果输入信息是假的呢?

面向技术内行人士

IEEE Spectrum 是 IEEE 的旗舰出版物,而 IEEE 是全球最大的致力于工程与应用科学的专业组织。我们的文章、视频和信息图表让读者了解技术、工程和科学领域的最新进展。

创建账户,即可享受更多免费内容和权益

保存文章以便稍后阅读需要 IEEE Spectrum 账户

The Institute 的内容仅会员可见

下载完整 PDF 期刊仅限 IEEE 会员

下载本电子书仅限 IEEE 会员

访问 Spectrum 数字版仅限 IEEE 会员

关注主题是仅限 IEEE 会员的功能

加入全球最大的致力于工程与应用科学的专业组织,即可访问 Spectrum 的所有文章、档案、PDF 下载及其他权益。进一步了解 IEEE →

加入全球最大的致力于工程与应用科学的专业组织,即可获取本电子书以及 IEEE Spectrum 的所有文章、档案、PDF 下载及其他权益。进一步了解 IEEE →

访问数千篇文章——完全免费

创建账户即可获取专属内容与功能:保存文章、下载合集并发表评论——全部免费!如需完整访问权限与权益,请订阅 Spectrum。

你的机器人的安全功能已经能正常工作。但如果输入在说谎呢?

VicOne 如何帮助揭示 AI 机器人防护机制中的网络风险

VicOne 是一家物理 AI 网络安全公司,致力于保护车辆和机器人免受可能改变其行为并危及安全的网络威胁。

机器人可以遵守自身的安全规则,却仍然依据被操纵的信息行事。来看看这对机器人安全意味着什么,以及团队如何加强测试与证据。

本文由 VicOne 呈现。

一台移动机器人在走廊里为某人减速。一只协作机械臂在工人靠近时放缓动作。一台人形机器人停下来让人通过。每一次响应都取决于关于机器人自身或周围环境的信息:一个距离读数、一个位置估计,或一个停止信号。

但如果其中某个输入是错的,而机器人仍然完全按设计做出响应,会发生什么?移动机器人把走廊判定为畅通,而那里仍有人。机械臂估计工人离得更远。人形机器人没有在人横穿其路径时收到停止信号。

在每种情况下,故障都可能造成这种不匹配。蓄意操纵同样可能。

在 VicOne LAB R7,我们研究被操纵的输入如何改变机器人行为——以及这对旨在保障人身安全的系统意味着什么。

不可信输入如何改变机器人的行为

机器人所做的远不止测量周围环境。它可能会解读所见所闻,以决定下一步做什么。因此,放置在其环境中的信息可以影响它的行为。

已发表的研究显示了这种情况如何发生。在一项针对视觉-语言-动作(VLA)模型的研究中,摄像头视野中的一个贴片降低了模拟机器人测试中的任务成功率。FreezeVLA 发现,一张对抗性图像可导致受测模型忽略后续指令。尽管这些研究采用了不同方法,但两者都表明视觉输入会如何干扰机器人既定的任务。

在一项使用 Gemma 4 E4B 的机器狗测试中,海报上的文字被当作指令处理,改变了机器人的运动。

VicOne LAB R7 也测试了不可信输入如何改变机器人行为。在一项使用 Gemma 4 E4B 的机器狗测试中,海报上的文字被当作指令处理,改变了机器人的运动。在另一项使用 Nemotron、运行于 NVIDIA Jetson AGX Orin 的医院服务机器人模拟中,人耳听不见的精心构造音频改变了机器人的模拟行为。分配的任务并未改变,改变的是输入。

在 VicOne LAB R7 的一项测试中,海报上的文字改变了机器狗的运动,说明精心构造的输入如何能把一个不太可能的结果变成可重复的攻击。VicOne

与安全相关的决策同样有输入

独立的防护系统或许仍能阻止危险动作。但该系统同样依赖信息:告诉它何时介入的读数、估计值和消息。如果其中某一项输入被操纵,会发生什么?

在一场机器人漏洞赏金活动中,VicOne LAB R7 的研究人员向一台机器人注入了 ROS 2/DDS 消息,而主办方原本预期该机器人在安全控制设置下会保持静止。机器人却动了。

人形机器人的平衡控制器是另一个例子。如果它依赖一个存在漏洞的陀螺仪,那么以该传感器谐振频率发出的声音可能会扭曲其上报的旋转数据。控制器可能会针对一个从未发生的倾斜进行校正。研究人员曾演示了针对陀螺仪存在漏洞的无人机的底层声学攻击。该研究并未演示同样的攻击链导致人形机器人摔倒。

其他防护措施可以中断其中任一环节。对安全与安保团队而言,问题在于这些防护是独立检测到危险,还是依赖同样被操纵的信息。

图 1。VicOne 梳理了意外故障与蓄意网络操纵如何导致相似的机器人安全后果,从失去控制到人身伤害。VicOne

蓄意攻击挑战安全假设

安全评估可能会认为意外故障的组合不太可能发生。蓄意操纵改变了这一假设。攻击者可以选择何时引入虚假输入,并重复触发同一条件。这并不使伤害变得不可避免,但可能改变网络风险应如何评估。

冗余传感器也需要进行对抗性测试。如果某一个动作能够同时影响两路输入,它们的一致性所能提供的保障可能低于预期。在自动驾驶研究中,一个精心构造的物理物体误导了结合摄像头与 LiDAR 感知的系统。该研究并未证明某一特定机器人存在弱点,但它说明了为什么团队应当测试:一次操纵是否能够同时影响多项检查。

网络威胁为安全考量加入了意图因素,因此基于意外故障的风险评估必须重新审视,以纳入能够被择时并重复实施的攻击。

安全测试强化机器人安全证据

安全团队会为速度、防护距离和允许作业区域设定限值。这些限值背后的证据还应说明:当用于执行限值的信息被蓄意操纵时,会发生什么。

这些证据还可以支持针对适用要求的评估。中国的 GB/T 45502-2025 涉及服务机器人的信息安全。IEC TS 63074 审视可能影响安全相关控制系统的信息安全威胁。欧盟《机械法规》自 2027 年 1 月 20 日起适用,其中包含保护安全相关系统和数据免遭破坏的要求。每项标准各有其适用范围,团队仍需针对其机器人的设计和运行条件提供证据。

在部署之前,安全工程师与信息安全工程师可以对照相同的安全限值,比较正常场景与对抗性场景。在运行期间,他们可以监测软件、模型、传感器信号或行为的变化,这些变化可能使早先的结果受到质疑。对可疑行为的任何响应都应遵循安全团队批准的策略。

VicOne 为覆盖机器人全生命周期的工作提供支持。通过 Robotic Hacking Community,VicOne LAB R7 与研究人员合作,调查网络威胁如何改变机器人的行为。VicOne 的 Radeis 帮助团队在部署前验证潜在影响,而 Rthena 则在运行期间提供风险与行为的可见性。

安全与安保团队应共同回答的五个问题

核心考验在于:当机器人所依赖的信息被操纵时,其防护机制是否仍能保护人。安全与安保团队可以先从追溯每项防护决策背后的输入入手,对其提出质疑,并随着机器人的变化重新审视这些证据。他们可以从以下五个问题开始:

1. 每项防护功能读取什么?梳理各类读数、估计值、消息和确认信号。找出哪些防护机制独立于机器人的 AI 决策运行。

2. 这些信息是如何到达的?检查输入是如何产生、传输、在适当情况下进行身份认证的,以及当输入缺失或不合常理时如何处理。

3. 如果某个输入被操纵会怎样?针对机器人的安全限值,测试虚假距离、漂移的位置估计以及其他相关场景。

4. 一个动作能否误导多个输入?检查传感器融合或看似独立的防护机制是否共享同一失效点。

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Your Robot’s Safety Functions Already Work. What If the Input Lies?)

Inside Robot Safety and Cybersecurity for AI Robots - IEEE Spectrum

The November issue of IEEE Spectrum is here!

Your Robot’s Safety Functions Already Work. What If the Input Lies?

FOR THE TECHNOLOGY INSIDER

IEEE Spectrum is the flagship publication of the IEEE — the world’s largest professional organization devoted to engineering and applied sciences. Our articles, videos, and infographics inform our readers about developments in technology, engineering, and science.

Enjoy more free content and benefits by creating an account

Saving articles to read later requires an IEEE Spectrum account

The Institute content is only available for members

Downloading full PDF issues is exclusive for IEEE Members

Downloading this e-book is exclusive for IEEE Members

Access to Spectrum 's Digital Edition is exclusive for IEEE Members

Following topics is a feature exclusive for IEEE Members

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to this e-book plus all of IEEE Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →

Access Thousands of Articles — Completely Free

Create an account and get exclusive content and features: Save articles, download collections, and post comments — all free! For full access and benefits, subscribe to Spectrum .

Your Robot’s Safety Functions Already Work. What If the Input Lies?

How VicOne helps expose cyber risks in AI robot safeguards

VicOne is a Physical AI cybersecurity company protecting vehicles and robots from cyber threats that can alter behavior and compromise safety.

Humanoid robot and robot dog meet a woman on a sunny terrace at sunset

A robot can follow its safety rules and still act on manipulated information. See what this means for robot safety and how teams can strengthen their testing and evidence.

This article is brought to you by VicOne.

A mobile robot slows for someone in a hallway. A collaborative arm eases its movement as a worker approaches. A humanoid pauses to let a person pass. Each response depends on information about the robot or its surroundings: a distance reading, a position estimate, or a stop signal.

But what happens when one of those inputs is wrong, and the robot still responds exactly as designed? The mobile robot reads the hallway as clear while someone is still there. The arm estimates that the worker is farther away. The humanoid does not receive a stop signal as a person crosses its path.

In each case, a fault could cause that mismatch. So could deliberate manipulation.

At VicOne LAB R7, we examine how manipulated inputs can change robot behavior — and what that means for the systems meant to keep people safe.

How an untrusted input can change a robot’s behavior

A robot does more than measure its surroundings. It may interpret what it sees and hears to decide what to do next. Information placed in its environment can therefore influence its actions.

Published research shows how this can happen. In a study of vision-language-action (VLA) models, a patch in a camera’s view reduced task success in simulated robot tests. FreezeVLA found that an adversarial image could cause tested models to ignore later instructions. Although the studies used different methods, both demonstrate how visual input can interfere with a robot’s intended task.

In a robot-dog test using Gemma 4 E4B, text on a poster was treated as an instruction and changed the robot’s movement.

VicOne LAB R7 also tested how untrusted inputs could change robot behavior. In a robot-dog test using Gemma 4 E4B, text on a poster was treated as an instruction and changed the robot’s movement. In a separate hospital-service-robot simulation using Nemotron on NVIDIA Jetson AGX Orin, crafted audio that is inaudible to people changed the robot’s simulated behavior. The assigned tasks did not change; the inputs did.

In a VicOne LAB R7 test, text on a poster changed a robot dog’s movement, illustrating how a crafted input can turn an unlikely outcome into a repeatable attack.VicOne

Safety-related decisions have inputs too

An independent protective system may still stop a dangerous action. But that system also depends on information: readings, estimates, and messages that tell it when to intervene. What happens if one of those inputs is manipulated?

At a robotics bug bounty event, VicOne LAB R7 researchers injected a ROS 2/DDS message into a robot that organizers expected to remain still under a safe-control setting. The robot moved.

A humanoid’s balance controller offers another example. If it relies on a vulnerable gyroscope, sound at the sensor’s resonant frequency could distort its reported rotation. The controller might correct for a tilt that never occurred. Researchers demonstrated the underlying acoustic attack against drones with vulnerable gyroscopes. The research did not demonstrate the same attack chain causing a humanoid to fall.

Other protections could interrupt either chain. The question for safety and security teams is whether those protections detect the danger independently or depend on the same manipulated information.

Table comparing AI safety risks (bugs, failures) with security risks (attacks, hijacks).Figure 1. VicOne maps how accidental faults and deliberate cyber manipulation can lead to similar robot safety outcomes, from loss of control to physical harm.VicOne

Deliberate attacks challenge safety assumptions

A safety assessment may treat a combination of accidental faults as unlikely. Deliberate manipulation changes that assumption. An attacker can choose when to introduce a false input and repeat the same trigger. That does not make harm inevitable, but it may change how cyber risks should be assessed.

Redundant sensors also need an adversarial test. If one action can affect both inputs, their agreement may offer less reassurance than expected. In autonomous driving research, a crafted physical object misled a system combining camera and LiDAR perception. The study does not establish a weakness in a particular robot, but it shows why teams should test whether one manipulation can affect several checks at once.

Cyber threats add intent to the safety equation, so risk assessments based on accidental faults must be revisited to account for attacks that can be timed and repeated.

Security testing strengthens robot safety evidence

Safety teams set limits for speed, protective distances, and permitted operating areas. The evidence behind those limits should also show what happens when the information used to enforce them is deliberately manipulated.

That evidence can also support assessments against applicable requirements. China’s GB/T 45502-2025 addresses information security for service robots. IEC TS 63074 examines security threats that could affect safety-related control systems. The EU Machinery Regulation, which applies from January 20, 2027, includes requirements to protect safety-relevant systems and data against corruption. Each has its own scope, and teams still need evidence for their robot’s design and operating conditions.

Before deployment, safety and security engineers can compare normal and adversarial scenarios against the same safety limits. During operation, they can monitor for changes to software, models, sensor signals, or behavior that may call earlier results into question. Any response to suspicious behaviors should follow policies approved by the safety team.

VicOne supports that work across the robot lifecycle. Through the Robotic Hacking Community, VicOne LAB R7 works with researchers to investigate how cyber threats can change robot behavior. VicOne’s Radeis helps teams validate potential effects before deployment, while Rthena provides visibility into risks and behavior during operation.

Five questions safety and security teams should answer together

The central test is whether a robot’s safeguards still protect people when the information they rely on is manipulated. Safety and security teams can begin by tracing the inputs behind each protective decision, challenging them, and revisiting the evidence as the robot changes. They can start with these five questions:

1. What does each protective function read? Map the readings, estimates, messages, and confirmations. Identify which safeguards operate independently of the robot’s AI decisions.

2. How does that information arrive? Check how inputs are produced, transmitted, authenticated where appropriate, and handled when they are missing or implausible.

3. What happens if an input is manipulated? Test false distances, drifting position estimates, and other relevant scenarios against the robot’s safety limits.

4. Can one action mislead several inputs? Check whether sensor fusion or seemingly independent safeguards share a point of failure.

5. When should the evidence be revisited? Reassess after changes to software, models, sensors, or operating conditions. Agree on bounded responses to suspicious behavior with the safety team in advance.

The goal is not only to show that a protective function works. Teams also need evidence that it remains protective when the information behind its decision is wrong or deliberately manipulated — and that another safeguard can catch the resulting hazard when needed. As robots change, that evidence needs to change with them.

For a deeper look how cybersecurity can affect robot safety, download our white paper “Why Robot Safety Depends on Cybersecurity: Protecting the Information That Safety Decisions Rely On.”

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 IEEE Spectrum AI(Your Robot’s Safety Functions Already Work. What If the Input Lies?),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗