NVIDIA GR00T 2 看人类视频想象自己成机器人

为了让机器人学会新任务,人类通常必须在训练期间费力地对其进行遥操作。NVIDIA 的新 AI 模型 GR00T 2 旨在通过将视频中的人类动作迁移到机械臂上,来减少这种工作量。
为了训练机器人完成新任务,工程师通常要花数小时远程控制这些机器。这种所谓的遥操作成本高昂、速度缓慢,而且每款机器人模型都需要自己的数据。在一份关于 GR00T 2 的新研究报告中,NVIDIA 介绍了一款旨在缩短这一过程的新 AI 基础模型。
与包括前代 GR00T 1 在内的以往方法不同,GR00T 2 不会将摄像头图像和语言指令直接转换为电机运动。NVIDIA 表示,该软件以该公司所称的“世界-动作模型”方式运作。它会同时计算下一个动作以及紧接未来的一小段视频。例如,系统会预测当机械臂拿起物体时,摄像头图像应如何变化。
与前代不同,GR00T 2 不再是视觉-语言-动作模型。NVIDIA 称其为世界-动作模型(WAM),它还会通过扩散模型预测未来将看到的内容。| 图片:NVIDIA
这种视频预测应能让模型在不同机器人形态之间迁移动作序列。NVIDIA 在一个拥有超过一千块 GB300 芯片的集群上训练了该系统,所用数据包括约 40 万小时、展示人们佩戴头戴式摄像头执行各种任务的视频素材。
研究人员表示,要让机器人学会一项新任务,只需一个人佩戴数据手套、专用手部夹爪或手指外骨骼来演示这些步骤即可。在训练过程中把这些记录与目标机器人的通用运动数据结合起来时,AI 在其视频预测中生成的是机械臂而非人的手臂,并由此推导出相匹配的控制指令。
在实验室测试中,一台配有五指手的机器人通过这种方式就能点燃火柴、切香蕉,或把包裹抛上传送带,而无需针对这些具体任务进行遥操作。
成功率因任务不同而在 10% 到 80% 之间波动。仅凭一段人手运动的视频,该系统还通过计算机模拟学会了钉钉子。在 50 项仅用少量示例后模型已表现良好的选定任务中,不到三分钟的传统训练就足以把成功率从 40% 提升到 95%。
更高层的语言模型有助于纠错,但会拖慢机器人
对于耗时更长的装配工作,Nvidia 将这套系统与更高层的 AI 语言模型配合使用,例如 OpenAI 的 GPT-6 Astra。GR00T 2 负责抓取动作,而语言模型则把任务拆解为一个个步骤,在摄像头图像中用颜色标出抓取点,并在零件掉落时启动辅助程序。
当研究人员把玩具模型的散件以任意顺序摆在桌上时,该组合系统在至少 97% 的试验中完成了组装。若没有用于纠错的语言模型和额外的微调,玩具汽车的组装在 100 次尝试中仅成功了 23 次。团队还通过在真实设备上反复试错,训练了诸如插网线或插充电器这类精细操作任务。
针对双足机器人,NVIDIA 开发了精简版 GR00T 2 Turbo,它处理的图像细节更少。该模型直接运行在人形机器人的板载计算机 Jetson Thor 上。在测试中,这台机器能够舀爆米花、清理餐桌,并用脚踩的毛巾擦干液体。不过最后一项任务在所有尝试中的成功率略低于一半。
研究团队指出,该系统要想在真实环境中可靠应用,还有重大障碍需要跨越。该模型目前尚不能直接处理触觉传感器数据,有时会漏掉诸如螺丝只拧了一半这类细微错误。上层的语言模型也显著拖慢了速度。在垃圾分拣过程中,机器人约有三分之一的总时间处于闲置状态,因为它平均需要等待超过 7 秒,才能得到规划模型的下一次计算结果。
其他实验室也在寻找用更少遥操作来教机器人学习新任务的方法。我们在关于 Neolabs 的 AI Radar #5 中介绍过其中一些。
没有炒作的 AI 新闻——由人类精选
完整访问 THE DECODER 上的每一篇文章
参与评论和社区讨论
每周通过邮件发送的 AI 新闻回顾
每年 6 次:"AI Radar"——深入解读最重要的 AI 议题
每日 AI 新闻,始终紧跟最新动态
我们完整的十年存档
由一支拥有 10 年以上 AI 经验的团队负责报道
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Nvidia's model GR00T 2 watches humans on video, then imagines itself as the robot)
To teach robots new tasks, humans usually have to laboriously teleoperate them during training. Nvidia's new AI model GR00T 2 is designed to cut that effort by transferring human movements from videos to robotic arms.
To train a robot for new tasks, engineers usually spend hours remotely controlling the machines. This so-called teleoperation is expensive, slow, and needs its own data for every robot model. In a new research report on GR00T 2, Nvidia introduces a new AI foundation model designed to shorten this process.
Unlike previous approaches, including its predecessor GR00T 1, GR00T 2 doesn't translate camera images and language commands directly into motor movements. Nvidia says the software works as what the company calls a world-action model. It simultaneously calculates the next movement and a short video of the immediate future. The system predicts how the camera image should change when a robotic arm picks up an object, for example.
Unlike its predecessor, GR00T 2 is no longer a vision-language-action model. Nvidia calls it a world-action model (WAM) that also predicts what it will see in the future via diffusion. | Image: Nvidia
This video prediction should let the model transfer movement sequences between different robot form factors. Nvidia trained the system on a cluster with more than a thousand GB300 chips, including about 400,000 hours of video footage showing people performing various tasks while wearing head-mounted cameras.
To then teach a robot a new task, the researchers say it should be enough for a person to demonstrate the steps using data gloves, specialized hand grippers, or a finger exoskeleton. When these recordings are combined during training with general motion data from the target robot, the AI generates the robotic arm instead of the human arm in its video prediction and derives the matching control commands from that.
In lab tests, a robot with five-fingered hands was able to light a match, slice a banana, or toss packages onto a conveyor belt this way, without having been teleoperated for those specific tasks.
Success rates varied between 10 and 80 percent depending on the task. From a single video of a human hand movement, the system also learned to hammer nails by way of a computer simulation. For 50 selected tasks where the model already performed well after just a few examples, less than three minutes of conventional training was enough to raise the success rate from 40 to 95 percent.
A higher-level language model helps with errors but slows the robot down
For longer assembly work, Nvidia pairs the system with a higher-level AI language model like OpenAI's GPT-6 Astra. While GR00T 2 handles the gripping movements, the language model breaks the task into individual steps, color-codes grip points in the camera image, and launches helper routines when a part falls off.
When the researchers placed loose parts for toy models on a table in no particular order, the combined system assembled them in at least 97 percent of trials. Without the corrective language model and additional fine-tuning, the toy car assembly succeeded in only 23 out of 100 attempts. The team also trained precision tasks like plugging in network cables or chargers through repeated trial and error on the real device.
For bipedal robots, Nvidia developed the slimmed-down variant GR00T 2 Turbo, which processes fewer image details. It runs directly on the Jetson Thor onboard computer of a humanoid robot. In testing, the machine was able to scoop popcorn, clear tables, and wipe up liquids with a towel under its foot. That last task, though, succeeded in just under half of all attempts.
The research team identifies significant hurdles before the system is ready for reliable real-world use. The model doesn't yet process direct sensor data for touch and sometimes misses subtle errors like half-turned screws. The higher-level language model also slows things down considerably. During trash sorting, the robot stood idle for roughly a third of the total time because it had to wait more than seven seconds on average for the planning model's next computation.
Other labs are also looking for ways to teach robots new tasks with less teleoperation. We covered some of them in our AI Radar #5 on Neolabs.
AI News Without the Hype – Curated by Humans
Full access to every article on THE DECODER
Join the comments and community discussions
A weekly AI news recap via mail
6x/year: "AI Radar" — deep dives on the AI topics that matter most
Daily AI news, always up to date
Our full ten-year archive
Covered by a team with 10+ years in AI
原文为英文,正文由自动抓取并保留原文。
原文发布于 The Decoder(Nvidia's model GR00T 2 watches humans on video, then imagines itself as the robot),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。