推出多模态知识智能体基准测试

高级产品经理,Google DeepMind
研究总监,Google DeepMind
你的浏览器不支持音频元素。
我们与 Artificial Analysis 合作推出 MKA(Multimodal Knowledge Agent,多模态知识智能体),这是一个全新的基准测试,用于评估智能体如何跨视觉复杂的文件进行推理,以完成真实世界的知识工作类任务。
大多数知识工作类基准测试仅凭文本就能完成,根本无需查看图像或页面布局。然而,日常专业工作的实际样貌与此大不相同。在许多行业中,日常工作高度依赖视觉信息——从交叉比对建筑蓝图、追踪工程图纸,到把研究结论整合成精美的演示文稿、报告和电子表格。我们构建 MKA,就是为了衡量智能体在处理这类视觉复杂工作时的表现。
为了让这一基准测试扎根于真实工作流程,我们与 10 个垂直行业、83 个职业领域的专家合作,这些领域的日常工作天然具有视觉属性。从建筑师、生物学家到核保师、数据科学家,这些专家共同打造了 MKA 的 200 项任务评估集,用来辅助他们完成自己的工作。
每项任务都对应一个多步骤项目,资深专业人士需要三到八小时才能完成;任务将一条贴近真实的提示词和一整套多模态输入文件,与专家撰写的评分细则相配对,后者包含约 70 项“是/否”检查项,其中事实准确性的权重为 75%,视觉设计的权重为 25%。
三个 MKA 任务示例,涵盖从提示词、输入文件到标准输出和评分细则的全过程。
从今天起,你可以在 Artificial Analysis 上探索该基准测试,并评估你自己的模型。每个模型都在同一个开源智能体运行框架 Stirrup 中运行,使用相同的工具、沙箱和轮次预算。
一个基于非 Google 模型构建的智能体式自动评分器会为每项交付成果打分。它会打开每个文件、渲染页面、检查电子表格公式,并依据专家评分标准核对工作成果。当领域专家亲自为模型输出评分时,他们与自动评分器的一致率达到 96%。为确保结果有意义,Artificial Analysis 运行包含 200 项任务的评估集,并将其保密,以免这些任务泄露到训练数据中,或被模型通过搜索找到。我们还在 Hugging Face 上发布了另一组 10 项公开任务,以便任何人都能查看格式并测试自己的智能体。我们的技术报告详细介绍了方法论和结果。它表明前沿模型的工作方式差异很大。在相同任务上,我们测试的模型在效率上存在极大差异,平均每个任务需要 30 到 130 轮才能完成。我们展示了强大的感知能力对于解决每项任务为何必要,也展示了当移除视觉能力时,模型如何尝试弥补却最终失败。
以 Gemini 4 Carbon 拓展视觉前沿
在我们测试的模型中,Gemini 4 Carbon 的质量排名第 2,并且相比得分最高的模型,平均完成任务成本低 6 倍。
在 MKA 评估中,Gemini 4 Carbon 处于质量与价格的帕累托前沿。
Gemini 4 Carbon 今日起可通过 Google AI Studio 中的 Gemini API 使用。探索我们更新的 Gemini 4 开发者指南和文档,开始使用。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Introducing the Multimodal Knowledge Agent benchmark)
Senior Product Manager, Google DeepMind
Research Director, Google DeepMind

Your browser does not support the audio element.
In partnership with Artificial Analysis, we're launching MKA (Multimodal Knowledge Agent), a new benchmark built to evaluate how agents reason across visually complex files to solve real-world knowledge work tasks.
Most knowledge work benchmarks can be solved with text alone, without ever needing to look at an image or page layout. Yet, everyday professional work looks very different. Across many industries, day-to-day work is deeply visual — from cross-referencing architectural blueprints and tracing engineering schematics to synthesizing findings into polished decks, reports, and spreadsheets. We built MKA to measure how well agents can handle this visually complex work.
To ground the benchmark in real-world workflows, we partnered with domain experts across 10 verticals and 83 occupations where day-to-day work is inherently visual. Together, these specialists — from architects and biologists to underwriters and data scientists — created MKA's 200-task evaluation set to assist them in their jobs.
Each task reflects a multi-step project that takes a seasoned professional three to eight hours to complete, pairing a realistic prompt and a bundle of multimodal input files with an expert-written rubric of roughly 70 yes/no checks, weighted 75% on factual accuracy and 25% on visual design.
Three MKA tasks, from prompt and input files to golden output and rubric.
Starting today, you can explore the benchmark and evaluate your own models on Artificial Analysis. Every model runs in the same open-source agent harness, Stirrup, with the same tools, sandbox, and turn budget.
An agentic autorater built on a non-Google model grades each deliverable. It opens every file, renders pages, inspects spreadsheet formulas, and checks the work against the expert rubric. When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time.
To keep results meaningful, Artificial Analysis runs the 200-task evaluation set and keeps it private so the tasks can't leak into training data or be found by the model via search. We've also released a separate set of 10 public tasks on Hugging Face, so anyone can inspect the format and test their own agents.
Our technical report covers the methodology and results in detail. It shows that frontier models work in very different ways. On the same tasks, the models we tested have extreme differences in efficiency, using between 30 and 130 turns on average to complete each task. We show how strong perception abilities are necessary to solve each task, and we also show how models try and fail to compensate when vision is removed.
Expanding visual frontiers with Gemini 4 Carbon
Gemini 4 Carbon achieves #2 in quality among the models we tested, and completes tasks at 6x lower cost on average compared to the top scoring model.
Gemini 4 Carbon is on the pareto frontier of quality and price for the MKA eval.
Gemini 4 Carbon is available today via the Gemini API in Google AI Studio. Explore our updated Gemini 4 developer guide and documentation to get started.
原文为英文,正文由自动抓取并保留原文。
原文发布于 Google Blog(Introducing the Multimodal Knowledge Agent benchmark),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。