AI 助数字人说话时动作更自然

编辑:Sadie Harley,审阅:Robert Egan
本文已按照 Science X 的编辑流程与政策完成审阅。编辑在确保内容可信度的同时,着重标注了以下属性:
经同行评议的出版物
添加到首选来源
DynaConTalk 在对齐的小波空间中,将多模态语音线索映射为整体的 3D 伴随语音动作。来源:arXiv DOI: 10.48550/arxiv.2610.09846
人们交流时用到的不只是语言。举起的手、姿态的变化或一个面部表情,都有助于传达含义与情绪。如今,研究人员开发出一套人工智能(AI)系统,能够把语音转化为身体、手部和面部的协调动作,有望让数字角色和虚拟主播更具表现力、更加逼真。
DynaConTalk 由东北大学和香港大学的研究人员开发,可生成与所说内容相对应的三维(3D)人体动作。
与仅生成单个手势的方案不同,该系统能协调身体的不同部位,同时在较长的演讲过程中保持动作的流畅。用户还可以通过指定姿态或提供参考示例来引导生成的动作。
该研究成果已发表在 ACM Transactions on Graphics 上。这项研究还计划在 SIGGRAPH Asia 2026 上展示,该会议将于 12 月 1 日至 4 日在马来西亚吉隆坡举行。
DynaConTalk 根据语音生成的一系列姿势。不断变化的体态、手势和面部表情展示了该系统如何随时间协调身体各部位的动作。图片来源:Yifei Zhu 等人。
协调大幅与细微动作
根据语音生成逼真的全身动作面临多项挑战。身体姿态的大幅变化会掩盖手指手势、面部表情等更细微的动作,使 AI 模型难以学习,从而可能导致手势重复或出现不自然的抖动。另一个问题出现在将分别生成的动作片段拼接在一起时,有时会产生明显的速度或节奏变化。
为解决这些问题,DynaConTalk 将动作拆分为较慢和较快的分量,同时保持二者同步。这类似于将声音拆分为低频和高频分量。
通过同时学习这些不同类型的动作,该系统既能捕捉姿态的大幅变化,也能捕捉细微、短暂的手势。它还会在生成动作时动态调整对不同语音特征(包括节奏、声音特征和所说词语)的依赖程度。
该系统还利用此前生成的动作信息来引导下一片段。这使其能够在较长的序列中保持连贯性,减少突兀的过渡,并有助于在长时间演讲中生成更自然的动作。
在四个类别中获得更高评分
研究人员使用两个公开可用的语音与人体动作数据集 BEAT2 和 SHOW 对 DynaConTalk 进行了评估。他们还开展了一项研究,让 30 名参与者对三个不同系统基于 15 个语音示例生成的视频进行比较。
在自然度、与说话内容的一致性、同步性和动作多样性这四个评估类别中,DynaConTalk 均获得了最高的平均评分。自动化评估进一步显示,其生成的动作与所记录的人体动作分布在分布上更为接近。
从动画角色到行走
这项技术可应用于电子游戏和电影的角色动画、虚拟主持人、在线教育和远程通信。例如,创作者可以指定角色的初始姿态,或提供一段参考动作,然后让 AI 生成周边的手势和表情。这提供了一种更灵活的方式来创建动画角色,使其动作与所说内容相得益彰。
该团队希望将这一方法扩展到更广泛的全身体动作,包括行走。然而,现有数据集包含的大幅动作样本相对有限,而仅凭语音也无法提供足够信息来确定一个人应如何在空间中移动。研究人员计划探索更大的动作数据集,以及关于移动路径的额外引导信息,以解决这些局限。
Yifei Zhu 等,DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion,ACM Transactions on Graphics (2026)。DOI: 10.1145/3842553。On arXivDOI: 10.48550/arxiv.2610.09846
期刊信息:ACM Transactions on Graphics
这篇报道的幕后是谁?
生命科学与生态学学士。拥有微生物学实验室背景,并在石油、天然气和可再生能源行业的医药新闻领域有从业经验。完整简介 →
数学生物学学士,创意写作硕士。游历广泛,对科学与语言有独特见解。完整简介 →
引用:AI 帮助数字人在说话时动作更自然(2026 年 11 月 13 日),2026 年 11 月 13 日检索自 https://techxplore.com/news/2026-11-ai-digital-humans-naturally.html
本文件受版权保护。除为私人学习或研究目的的合理使用外,未经书面许可,不得复制任何部分。内容仅供信息参考之用。
一种从语言提示生成人体动作的新框架
用 AI 驱动的程序创建逼真的“说话头像”
计算框架简化了 3D 环境中人体角色的合成动作
动态模型可生成逼真的人体动作并编辑现有动作
一款能够生成逼真说话人脸动画的模型
采用新型电解质设计,水系电池可在碘中储存更多能量
隐藏的守门人:微小岩石孔隙中的水如何塑造地下储能
新型 AI“copilot”让工程师在算法生成结构设计的过程中进行引导
受物理学启发的 AI 通过分析颜色与亮度模式来检测虚假人脸
可变形生物纱线为可穿戴技术编织新可能
当被告知正被监视时,AI 模型可能给出不同回答
获取即时摘要文本(Gist)
DynaConTalk 通过同步慢速与快速运动分量并连接连续片段,从语音生成协调的 3D 身体、手部和面部动作。在一项对三套系统的 30 人对比评测中,它在自然度、与语音的对应性、同步性和动作多样性方面获得了最高的平均评分。
本摘要由 LLM 自动生成。完整免责声明
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(AI helps digital humans move more naturally as they speak)
edited by Sadie Harley, reviewed by Robert Egan
This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:
peer-reviewed publication
Add to Preferred Sources
DynaConTalk maps multimodal speech cues to holistic 3D co-speech motion in an aligned wavelet space. Credit: arXiv DOI: 10.48550/arxiv.2610.09846
People communicate with more than words. A raised hand, a change in posture or a facial expression can help convey meaning and emotion. Now, researchers have developed an artificial intelligence (AI) system that can translate speech into coordinated movements of the body, hands and face, potentially making digital characters and virtual presenters more expressive and lifelike.
Developed by researchers at Tohoku University and the University of Hong Kong, DynaConTalk generates three-dimensional (3D) human movements that correspond to spoken content.
Unlike approaches that generate only individual gestures, the system coordinates different parts of the body while maintaining smooth movement over longer speeches. Users can also guide the generated motion by specifying poses or providing reference examples.
The findings were published in ACM Transactions on Graphics. The research is also scheduled to be presented at SIGGRAPH Asia 2026, which will take place in Kuala Lumpur, Malaysia, from December 1 to 4.
A sequence of poses generated by DynaConTalk from speech. The changing posture, hand gestures and facial expressions illustrate how the system coordinates movement across the body over time. Credit: Yifei Zhu et al.
Coordinating broad and subtle movements
Generating realistic full-body movement from speech presents several challenges. Large changes in body posture can overshadow smaller movements, such as finger gestures and facial expressions, making them difficult for AI models to learn. This can result in repetitive gestures or unnatural shaking. Another problem occurs when separately generated movement segments are joined together, sometimes creating noticeable changes in speed or rhythm.
To address these issues, DynaConTalk separates movement into slower and faster components while keeping them synchronized. This is similar to separating sound into low- and high-frequency components.
By learning these different types of movement together, the system can capture both broad changes in posture and subtle, brief gestures. It also dynamically adjusts how much it relies on different features of speech, including rhythm, voice characteristics and spoken words, as it generates movement.
The system also uses information from previously generated movements to guide the next segment. This allows it to maintain continuity across longer sequences, reducing abrupt transitions and helping produce more natural movement during extended speeches.
Credit: Tohoku University
Higher ratings across four categories
The researchers evaluated DynaConTalk using two publicly available data sets of speech and human movement, BEAT2 and SHOW. They also conducted a study in which 30 participants compared videos generated by three different systems across 15 speech examples.
DynaConTalk received the highest average ratings in all four evaluation categories: naturalness, correspondence with spoken content, synchronization and movement diversity. Automated evaluations further showed that its generated movements more closely resembled the distribution of recorded human movements.
From animated characters to walking
The technology could have applications in character animation for video games and films, virtual presenters, online education and remote communication. For example, creators could specify a character's initial pose or provide a reference movement, then let the AI generate the surrounding gestures and expressions. This offers a more flexible way to create animated characters whose movements complement what they are saying.
The team hopes to extend the approach to a wider range of whole-body movements, including walking. However, existing data sets contain relatively limited examples of large-scale movement, while speech alone does not provide enough information to determine how a person should move through space. The researchers plan to explore larger motion data sets and additional guidance about movement paths to address these limitations.
Yifei Zhu et al., DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion, ACM Transactions on Graphics (2026). DOI: 10.1145/3842553. On arXivDOI: 10.48550/arxiv.2610.09846
Journal information:ACM Transactions on Graphics
Who's behind this story?
BSc Life Sciences & Ecology. Microbiology lab background with pharmaceutical news experience in oil, gas, and renewable industries. Full profile →
Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →
Citation: AI helps digital humans move more naturally as they speak (2026, November 13) retrieved 13 November 2026 from https://techxplore.com/news/2026-11-ai-digital-humans-naturally.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
A new framework to generate human motions from language prompts
Creating realistic 'talking heads' with an AI-powered program
Computational framework simplifies synthesized motions for human characters in 3D environments
Dynamic model can generate realistic human motions and edit existing ones
A model that can create realistic animations of talking faces
Water-based batteries store more energy in iodine with new electrolyte design
The hidden gatekeeper: How water in tiny rock pores shapes underground energy storage
New AI 'copilot' lets engineers guide structural designs as algorithms create them
Physics-inspired AI detects fake faces by analyzing color and brightness patterns
Shape-shifting bioyarns weave new possibilities for wearable tech
AI models can answer differently when told they are being watched
Get Instant Summarized Text (Gist)
DynaConTalk generates coordinated 3D body, hand and facial movements from speech by synchronizing slow and fast motion components and linking successive segments. In a 30-person comparison of three systems, it received the highest average ratings for naturalness, correspondence with speech, synchronization and movement diversity.
This summary was automatically generated using LLM. Full disclaimer
This site uses cookies to assist with navigation, analyse your use of our services, collect data for ads personalisation and provide content from third parties. By using our site, you acknowledge that you have read and understand our Privacy Policy and Terms of Use.
原文为英文,正文由自动抓取并保留原文。
原文发布于 TechXplore(AI helps digital humans move more naturally as they speak),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。
