11-19 更新 实时
首页/ 算力/ 正文

唇读 AI 先识别最确定的词来补全句子

算力 来源:TechXplore 2026-11-13 20:00 0 阅读 机器翻译
唇读 AI 先识别最确定的词来补全句子

由 Swati Mestri 编辑,Andrew Zinin 审阅

本文已按照 Science X 的编辑流程与政策进行审阅。编辑在确保内容可信度的同时,重点强调了以下属性:

添加到首选来源

自回归解码与所提方法的概念对比。(a) 传统方法从左到右一次生成一个词,因此无法修正早期出现的错误,例如把“PACK”误识别为“BACK”。(b) 所提方法先找出可以确信识别的词,如“BAGS”和“TRIP”,再以这些词作为上下文,正确识别前面那个有歧义的词。图片来源:韩国科学技术院(KAIST)

仅凭唇部动作来理解语音十分困难,在嘈杂环境中尤其如此。韩国科学技术院(KAIST)的一个研究团队开发出一种 AI 技术,它先识别出可以高置信度确定的词,再利用周围上下文来消除句子中歧义部分的歧义。

这项技术有望实现不依赖声音的交流。例如,它可以把被噪声掩盖的语音转换为文本,或者帮助失聪或听障人士跟上对话。

由电气工程学院 Yong Man Ro 教授领导的研究团队开发出一项技术,可提升视觉语音识别(VSR)的准确率,该技术能将唇部动作转换为文本。团队采用了扩散大语言模型(DLLM),它通过反复填充被掩蔽的位置来生成句子。这样一来,模型就能利用出现在歧义唇部动作前后的词来进行解码。

该研究已发布在 arXiv 预印本服务器上。

优先解码最清晰的唇部动作

唇读之所以困难,是因为不同的发音可能产生非常相似的唇部动作。例如在英语中,/p/、/b/ 和 /m/ 仅凭视觉信息几乎无法区分。广泛使用的 AI 模型按固定顺序生成句子,从第一个词到最后一个词。因此,如果模型在句子前部选错了词,这一错误会影响其后的词语。

研究团队用优先解码最确定部分的方法取代了这种从首到尾的顺序。模型首先识别与清晰唇部动作对应的词语,然后把这些词作为上下文来解码其余位置。通过同时考虑句子中前部和后部的信息,模型逐步解决歧义位置。

两阶段训练与句子长度估计

研究人员还改进了训练过程,将其分为两个阶段。第一阶段,模型学习将视频中的唇部动作与对应文本对齐。第二阶段,模型学习对句子长度建模。

在解码过程中,模型根据视频时长估计出若干可能的句子长度,并为每个长度生成一个候选句子。随后,它根据各句子长度的可能性和模型对自身预测的置信度来评估这些候选,选出最可靠的句子。

该团队使用 LRS3——一个包含 433 小时唇读视频的数据集——作为带标注的训练数据来训练模型,这意味着这些视频都配有真值转录文本。该模型取得了 19.4% 的词错误率。词错误率衡量的是与正确转录文本相比,被错误识别、遗漏或添加的单词所占的比例。数值越低,表示性能越好。在仅使用 LRS3 作为带标注训练数据的现有方法中,这是已报告的最低错误率。

迈向无需声音的实时通信

在难以听清语音的场景下,这项技术可用于把唇部动作转换为文字的通信工具。例如,它可以帮助判断在机器轰鸣的工业现场,工人们说的是什么。它还可以被开发成支持失聪或听障人士交流的设备。

团队计划继续研究,缩小该 AI 模型的规模并降低其计算成本,以便它能部署到更广泛的设备上。另一个目标是让这项技术无需联网就能在可穿戴设备和机器人上实时运行。

“这项研究之所以有意义,是因为它表明,灵活控制 AI 生成单词的顺序,本身就能克服视觉歧义给语音识别带来的一个根本性局限,”Ro 表示。他补充说,这项研究将为物理 AI 奠定基础,使人们能够在日常生活和工业现场的各类设备上可靠地使用它。

KAIST 电气工程学院博士后研究员 Jeong Hun Yeo 博士为第一作者。博士生 Chae Won Kim 和 Hyeongseop Rha 为共同作者,Ro 为通讯作者。

Jeong Hun Yeo 等,《Diffusion Large Language Models for Visual Speech Recognition》,arXiv(2026)。DOI: 10.48550/arxiv.2605.28456

期刊信息:arXiv

这篇报道背后是谁?

Swati Mestri 拥有电子工程学士学位,自 2019 年起担任内容编辑。她曾编辑技术、医疗健康和材料科学领域的研究文档,尤其对技术和太空领域感兴趣。完整简介 →

拥有物理学硕士学位和科研经历。长期热衷科学新闻。在 Science X 的编辑工作中发挥关键作用。完整简介 →

引用:唇读 AI 通过优先识别最确定的词来补全句子中的空缺(2026 年 11 月 13 日),2026 年 11 月 13 日检索自 https://techxplore.com/news/2026-11-lip-ai-gaps-sentences-words.html

本文档受版权保护。除为私人学习或研究目的而进行的合理使用外,未经书面许可不得转载任何部分。内容仅供参考。

用于非接触式唇读与声学分析的综合数据集为语音识别技术带来潜力

下一代助听器或能透过口罩读唇

2 万个单词的视觉图谱揭示读唇者为何会混淆常见的相似词

AI 减少感官幻觉,即使在夜间或烟雾中也不例外

研究人员表明,读唇会激活与真实言语相似的大脑区域

隐藏的守门人:微小岩石孔隙中的水如何塑造地下储能

可变形的生物纱线为可穿戴技术编织新可能

可重构 MEMS 开关或能让芯片封装布线躲过 X 射线逆向工程

研究人员解开一个关于辐射的物理学谜题,或可催生更耐用的电子器件

在早期测试中,手势与震动腰带让无人机无需视觉反馈即可实现控制

电场可以改变半导体的散热方式

获取即时摘要文本(Gist)

一种扩散语言模型通过优先解码高置信度词,并利用双向上下文来消解唇部动作的歧义,从而提升唇读效果。它实现了19.4%的词错误率,是仅以LRS3作为标注训练数据的方法中所报告的最低水平。

本摘要由LLM自动生成。完整免责声明

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Lip-reading AI fills gaps in sentences by recognizing most certain words first)

edited by Swati Mestri, reviewed by Andrew Zinin

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources

KAIST Develops Lip-Reading AI That Decodes Clear Words First to Resolve Ambiguous Mouth Movements Conceptual comparison between autoregressive decoding and the proposed method. (a) The conventional approach generates words one at a time from left to right, so it cannot revise an early error, such as misrecognizing “PACK” as “BACK.” (b) The proposed method first identifies words it can recognize with confidence, such as “BAGS” and “TRIP,” and then uses them as context to correctly identify the ambiguous early word. Credit: The Korea Advanced Institute of Science and Technology (KAIST)

Understanding speech from lip movements alone is difficult, particularly in noisy environments. A research team at the Korea Advanced Institute of Science and Technology (KAIST) has developed an AI technology that first recognizes words it can identify with high confidence and then uses the surrounding context to resolve ambiguous parts of the sentence.

The technology is expected to enable communication without relying on sound. For example, it could convert speech obscured by noise into text or help people who are deaf or hard of hearing follow conversations.

A research team led by professor Yong Man Ro from the School of Electrical Engineering has developed a technology that improves the accuracy of visual speech recognition (VSR), which converts lip movements into text. The team used a diffusion large language model (DLLM), which generates a sentence by repeatedly filling in masked positions. This allows the model to decode ambiguous lip movements using words that appear both before and after them.

The study is posted on the arXiv preprint server.

Decoding the clearest lip movements first

Lip reading is challenging because different sounds can produce very similar lip movements. In English, for example, /p/, /b/ and /m/ are nearly indistinguishable based on visual information alone. Widely used AI models generate sentences in a fixed order, from the first word to the last. As a result, if the model selects an incorrect word early in a sentence, the error can affect the words that follow.

The research team replaced this first-to-last order with an approach that decodes the most certain parts first. The model first identifies words associated with clear lip movements and then uses those words as context to decode the remaining positions. By considering information from both earlier and later parts of the sentence, the model progressively resolves ambiguous positions.

Two-stage training and sentence length estimates

The researchers also improved the training process by dividing it into two stages. In the first stage, the model learns to align lip movements in the video with the corresponding text. In the second stage, it learns to model sentence length.

During decoding, the model estimates several plausible sentence lengths based on the video duration and generates a candidate sentence for each length. It then evaluates the candidates based on the likelihood of each sentence length and the model's confidence in its predictions, selecting the most reliable sentence.

The team trained the model using LRS3, a data set containing 433 hours of lip-reading video, as its labeled training data, meaning that the videos are paired with ground-truth transcripts. The model achieved a word error rate of 19.4%. Word error rate measures the proportion of words that are incorrectly recognized, omitted or added compared with the correct transcript. A lower value indicates better performance. This is the lowest error rate reported among existing methods that use only LRS3 as labeled training data.

Toward real-time communication without sound

The technology could be used in communication tools that convert lip movements into text in situations where speech is difficult to hear. For example, it could help determine what workers are saying at industrial sites with loud machinery. It could also be developed into devices that support communication for people who are deaf or hard of hearing.

The team plans to continue research on reducing the size and computational cost of the AI model so that it can be deployed on a wider range of devices. A further goal is to enable the technology to run in real time on wearable devices and robots without an internet connection.

"This study is meaningful because it demonstrates that flexibly controlling the order in which AI generates words can, by itself, overcome a fundamental limitation of speech recognition caused by visual ambiguity," said Ro. He added that the research would lay the foundation for physical AI that people can reliably use on a variety of devices in everyday life and at industrial sites.

Dr. Jeong Hun Yeo, a postdoctoral researcher in the School of Electrical Engineering at KAIST, is the first author. Ph.D. candidates Chae Won Kim and Hyeongseop Rha are co-authors, and Ro is the corresponding author.

Jeong Hun Yeo et al., Diffusion Large Language Models for Visual Speech Recognition, arXiv (2026). DOI: 10.48550/arxiv.2605.28456

Journal information:arXiv

Who's behind this story?

Swati Mestri

Swati Mestri holds a bachelor's degree in Electronics Engineering and has worked as a content editor since 2019. She has experience editing research documents across technology, health care, and materials science, and has a particular interest in technology and space. Full profile →

Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile →

Citation: Lip-reading AI fills gaps in sentences by recognizing most certain words first (2026, November 13) retrieved 13 November 2026 from https://techxplore.com/news/2026-11-lip-ai-gaps-sentences-words.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Comprehensive dataset for contactless lip reading and acoustic analysis holds potential for speech recognition tech

Next generation of hearing aids could read lips through masks

Visual map of 20,000 words reveals why lip-readers confuse common look-alikes

AI reduces sensory hallucinations, even at night or in smoke

Lip reading activates brain regions similar to real speech, researchers show

The hidden gatekeeper: How water in tiny rock pores shapes underground energy storage

Shape-shifting bioyarns weave new possibilities for wearable tech

Reconfigurable MEMS switches could hide chip package wiring from X-ray reverse engineering

Researchers solve a physics puzzle about radiation that could lead to tougher electronics

Hand gestures and vibrating belt enable drone control without visual feedback in early tests

Electric fields can change how semiconductors dissipate heat

Get Instant Summarized Text (Gist)

A diffusion language model improves lip reading by decoding high-confidence words first and using context from both directions to resolve ambiguous lip movements. It achieved a 19.4% word error rate, the lowest reported for methods using only LRS3 as labeled training data.

This summary was automatically generated using LLM. Full disclaimer

This site uses cookies to assist with navigation, analyse your use of our services, collect data for ads personalisation and provide content from third parties. By using our site, you acknowledge that you have read and understand our Privacy Policy and Terms of Use.

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 TechXplore(Lip-reading AI fills gaps in sentences by recognizing most certain words first),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗