AI 辅助学习平台助机器人超越字面指令推理

人类天生具有直觉。我们能感知所遇到字面词语之外的含义。如今,MIT 工程师开发的一种新的人工智能辅助学习平台,正让机器人能够从人类的自然语言指令中解读出更大的含义。
在本月即将举行的机器人学习年度会议(Conference on Robot Learning)上,来自 MIT 和纽约大学的研究人员将介绍 FLEX,这是一种用于机器人运动规划的灵活特征编辑流水线。
借助 FLEX,人类可以通过用自然、对话式语言给出指令来与机器人互动。机器人不仅能按字面遵循指令,还能从词语中深入解读,得出更广泛的概括,从而帮助改进其运动规划任务。
团队在多个模拟场景中演示了 FLEX,包括一个机器人端上一杯咖啡的场景。在这些虚拟场景中,一个人通过用自然语言输入指令与机械臂互动。当机器人将咖啡杯移过杂乱的桌子时,这个人可能会输入“不要洒到笔记本电脑上!”作为回应,机器人会推理为什么它应避开笔记本电脑。为此,FLEX 集成了一个大型语言模型(LLM)——这是一种在海量文本库上训练的 AI 系统,文本库包括社交媒体帖子、视频转录和网页内容,LLM 利用这些内容识别更宏观的趋势、模式和泛化规律。
“LLM 在互联网上见过太多文本,”研究作者、MIT 计算机科学与人工智能实验室(CSAIL)博士后 Nathan Dennler 说。“人们可能写过,‘哦,我的手机掉水里了,现在坏了’,或者‘我把咖啡洒到电脑上,现在它坏了’。所以 LLM 领会到液体加设备会导致东西不能正常工作。”
借助 LLM,机器人推理出它也应该避免撞到所有电子产品。在其更新后的路径中,机器人会调整路线以同时避开笔记本电脑和附近的手机,而无需那个人明确说明这一点。
“关键在于,你说的话背后存在某种更高层次的意图,”Dennler 说。“人们自然能理解这一点。但你必须仔细向机器人解释,这需要时间和精力。FLEX 提供了一个与机器人沟通的接口,让你可以说,‘当人们这样说时,他们的意思是那样。’”
这项研究由 MIT 航空航天学助理教授 Andreea Bobu 领导。共同作者包括 Audrey Lee SM ’26 和纽约大学的 Ilia Sucholutsky。
超越我们所说的话
在 MIT,Bobu 领导协作学习与自主研究(CLEAR)实验室,她的团队在那里开发自主智能体,这些智能体学习为人们工作、与人们一起工作,并在人们周围工作。她专注于开发辅助机器人和家用机器人,这些机器人能够执行物理任务,并实时推理和适应,以响应一个人自然、对话式的指令。
“机器人用数学来做轨迹规划。但我们不能指望消费者为了告诉机器人该做什么而去学这些数学,”Dennler 说,他是 CLEAR Lab 的成员。“与其让人们去学习如何以某种特定方式使用机器人,我们的目标是让人们通过自然语言与机器人交互。”
机器人学界正投入大量精力,把自然语言融入机器人学习流程。各种策略包括:用语言澄清任务目标、改变机器人的轨迹,以及更新它对何为安全路径的理解。这些现有方法都是用自然语言告诉机器人如何具体执行一项任务。但机器人其实有可能从我们告诉它的内容中学到更多东西。
Bobu 和她的同事正在拓展机器人能从自然语言指令中学到的东西。在他们的新研究中,他们聚焦于让机器人能够从有限的自然语言指令中,解读出一个人未明确说出的潜在偏好。
吸收常识
该团队的新机器人学习平台 FLEX 分为三步,帮助机器人从一条自然语言指令中读出更多信息,而不只是它被明确要求做的事。
第一步是“Add”(添加),它让机器人根据用户的自然语言指令,把一个特征加入其环境要素清单——这些要素与当前任务相关。例如,当机器人把一个保龄球从一个位置搬到另一个位置时,有人可能会插一句“别把它砸到我的马克杯上!”机器人随后就会把这只马克杯加入路径中需要避开的物品清单。第一步正是以往方法的做法。每当有人做出一次纠正,机器人就向自己的特征清单中添加内容,并更新轨迹以避开这些特征。
“当你的特征列表变得超级长,而且由于环境可能已经发生变化,其中一些特征不再相关时,问题就出现了,”Dennler 说。他指出,如果一个人必须逐一指明要避开的每一个具体特征,这种做法就会变得令人厌烦。
第二步“泛化(Generalize)”正是 FLEX 提供的新方案。这一步引入了一个大语言模型——在这里是 GPT-4o——它会把这个杯子,以及此人目前已经指定要避开的其他所有特征一并纳入考量。该 LLM 会自动在其庞大的数据储备中搜索这些特征之间的模式和关联,最终做出泛化判断:应当避开易碎物品,即便这个人并未明确说明这一点。
“大语言模型从互联网上见过来自形形色色的人和地方的如此多的文本,”Dennler 说。“由于它们在大量这类内容上接受过训练,其中一些常识就被 LLM 吸收了。”
在第三步“掩码(Mask)”中,机器人标准的计算机视觉系统会评估其环境,判断哪些物体已不再存在,然后把这些物体从特征列表中删除。整个三步流程大约耗时 5 秒。
该团队在机器人手臂执行三项不同任务的模拟中测试了 FLEX:端上一杯咖啡、移动一支蜡烛、搬运一个保龄球。在这三个场景中,志愿者都通过输入自然语言指令与虚拟机器人交互,例如“别洒到我的笔记本电脑上”“离那摞纸远点”“别掉到杯子上”。借助 FLEX,机器人不仅逐字执行了全部三条指令,还进行了更高层次的推理——不要洒到电子产品上、远离易燃物品、不要把货物掉在易碎物品上。
“每个人的家都不一样,每个人在意的事情也不一样,所以没有哪台机器人一开箱就知道什么对你重要,”Bobu 说。“FLEX 让我兴奋的地方在于,它只需一两句话就能领会这一点,而不用你逐条把规则讲清楚——而这正是人们勉强忍受的机器人与他们真正想放在家里的机器人之间的差别。”
这项工作部分得到了 MIT Generative AI Impact Consortium 和 MIT Schwarzman College of Computing 的 Social and Ethical Responsibilities of Computing 项目的支持。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(AI-assisted learning platform helps robots reason beyond literal commands)
Humans are intuitive by nature. We can sense meaning beyond the literal words that we encounter. Now, a new artificial intelligence-assisted learning platform developed by MIT engineers is enabling robots to read larger meaning into a human’s natural language commands.
In a study to be presented at the annual Conference on Robot Learning this month, researchers from MIT and New York University introduce FLEX, a flexible feature-editing pipeline for robot motion-planning.
With FLEX, humans can interact with a robot by giving it instructions in natural, conversational language. The robot can not only follow the instructions literally but also read into the words to draw larger generalizations that help to improve its motion-planning task.
The team demonstrated FLEX in several simulated scenarios, including one in which a robot serves a cup of coffee. In these virtual scenarios, a person interacts with a robotic arm by typing commands in natural language. As the robot moves the coffee cup over a cluttered table, the person might type “Don’t spill on the laptop!” In response, the robot will reason about why it should avoid the laptop. For this, FLEX incorporates a large language model (LLM) — a type of AI system that is trained on enormous libraries of text, including social media posts, video transcriptions, and web content, which the LLM uses to identify larger trends, patterns, and generalizations.
“LLMs have seen so many texts from the internet,” says study author Nathan Dennler, a postdoc in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “People might have written about, ‘Oh, my phone dropped in the water and now it doesn’t work,’ or ‘I spilled coffee on my computer and now it doesn’t work.’ So LLMs pick up on the idea that liquids plus devices leads to things not working.”
Through the LLM, the robot reasons that it should also avoid bumping into all electronics. In its updated path, the robot adjusts to avoid both the laptop and a nearby phone, without the person having to explicitly spell this out.
“The idea is that there is some higher-level intention behind the words that you say,” Dennler says. “People naturally get this. But you have to carefully explain that to a robot, which takes time and effort. FLEX gives you an interface of communication with a robot to say, ‘When people say this, they mean that.’”
The study was led by Andreea Bobu, assistant professor of aeronautics and astronautics at MIT. Co-authors are Audrey Lee SM ’26 and Ilia Sucholutsky of New York University.
Beyond what we say
At MIT, Bobu heads up the Collaborative Learning and Autonomy Research (CLEAR) Lab, where her group develops autonomous agents that learn to work for, with, and around people. She is focused on developing assistive and household robots that can carry out physical tasks, and reason and adapt in real-time, in response to a person’s natural, conversational commands.
“Robots use math to do their trajectory planning. But we can’t expect a consumer to learn to do this math in order to tell a robot what to do,” says Dennler, who is a member of the CLEAR Lab. “Instead of having to get people to learn how to use a robot in a specific way, the goal is to interact with the robot through natural language.”
There is a large effort in the robotics community to incorporate natural language into robotic learning pipelines. Various strategies include using language to clarify a task’s goal, redirect a robot’s trajectory, and update its understanding of what is a safe path to take. These existing methods use natural language to tell the robot how to physically perform a task. But a robot could potentially learn much more from what we tell it.
Bobu and her colleagues are expanding what robots can learn from natural language commands. In their new study, they focused on enabling a robot to interpret a person’s underlying, unspecified preference from limited natural language commands.
Absorbing common sense
The team’s new robot learning platform, FLEX, is a three-step process that helps a robot read more into a natural language command, beyond what it is explicitly told to do.
The first step, “Add,” instructs a robot to add a feature to its list of things in its environment that matter for the task at hand, based on a user’s natural language command. For instance, as a robot is moving a bowling ball from one location to another, a person might interject with “Don’t drop it on my mug!” The robot would then add the mug to a list of things to avoid in its path. This first step is how previous methods have worked. Every time a person makes a correction, the robot adds to its features list, and updates its trajectory to avoid those features.
“The problem happens when your feature list becomes super long, and some features are no longer relevant because your environment may have changed,” says Dennler, who notes that this approach can grow tiresome if a person has to specify every specific feature to avoid.
The second step, “Generalize,” is the new solution that FLEX provides. This step incorporates an LLM — in this case, GPT-4o — that considers the mug, along with any other features that the person has so far specified to avoid. The LLM automatically searches through its vast cache of data for patterns and connections among these features, to ultimately make a generalization that it should avoid objects that are fragile, even when the person has not explicitly said so.
“LLMs have seen so many texts from the internet, from all sorts of people and places,” Dennler says. “Because they’ve trained so much on those things, some of that common sense gets absorbed into the LLM.”
In the third step, “Mask,” the robot’s standard computer vision system assesses its environment to determine what objects are no longer present, and then deletes those objects from its features list. This entire three-step process takes about five seconds.
The team tested FLEX in simulations of a robot arm carrying out three different tasks: serving a cup of coffee, moving a candle, and carrying a bowling ball. In all three scenarios, volunteers interacted with the virtual robot by typing natural language commands, such as “Don’t spill on my laptop,” “Stay away from the stack of paper,” and “Don’t drop on the mug.” With FLEX, the robot followed all three commands literally and also reasoned at a higher level to not spill on electronics, stay away from flammable objects, and not drop its cargo on fragile things.
“Everyone’s homes are different, and everyone cares about different things, so no robot comes out of the box knowing what matters to you,” Bobu says. “What excites me about FLEX is that it can pick that up from a sentence or two instead of asking you to spell out every rule — and that’s the difference between a robot people put up with and one they actually want in their home.”
This work was partly supported by the MIT Generative AI Impact Consortium and the MIT Schwarzman College of Computing’s Social and Ethical Responsibilities of Computing program.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MIT News AI(AI-assisted learning platform helps robots reason beyond literal commands),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。