Genie One 准确率:精选智能体为何需要受治理的上下文

智能体和受治理层解决的是不同的问题。Metric Views、Domains、Subdomains 和 Pages 帮助 Genie One 回答那些原本要靠口口相传的经验知识才能回答的定义类问题,而面对需要从 PDF 等非结构化数据中找答案的问题时,就需要智能体。
每个业务度量只需在一个经过认证的 Metric View 中定义一次,并用一个 Page 说明它适用于什么情况。在另一项变更测试中,更新一个 Metric View 及其对应的 Page,就让所有 12 个相关的 Genie One 答案切换到了新定义,而无需对智能体做任何修改。
如果你去问财务和核保部门,2024 年到 2025 年赔付率是否改善了,你会得到两个相互矛盾但都正确的答案。在我们为这项测试搭建的一个虚构商业车险公司里,财务报告的是恶化 8.2 个百分点,核保报告的则是改善 1.5 个百分点。两者都没错。财务按日历年、以再保险后的净额口径统计,因此一笔较早赔案上不利的准备金发展会体现在今天的数字里。核保则严格按保单年度、以再保险前的总额口径并剔除巨灾,因为这是用于厘定费率的基准。而准备金部门依赖的又是完全不同的第三个指标:事故年最终赔付率。
SQL 可以完全正确,却仍然给出一个对提问者来说说不通的答案。难点在于:同一个术语存在多个有效定义时,要知道该套用哪一个业务定义,同时还要让共享的事实(例如已承保保单总数)保持一致。这正是我们想通过 Genie One 探究的问题。
Genie One 由 Genie Ontology(现已正式发布)提供支持,将这种业务上下文带入分析体验。Genie Ontology 从团队已经完成的工作中学习,自动从现有仪表板、查询、笔记本和语义模型中捕获业务逻辑、计算和关系,作为 Ontology Snippets。这种学习到的知识由建模知识补充:通过认证的 Metric Views、Domains 和 Subdomains 以及 Pages,在 Unity Catalog 中表示的明确定义、受治理的业务概念。在本研究中,我们将这种建模知识称为“治理层”。
学习到的知识和建模知识共同让 Genie 更深入地理解组织如何运作。团队可以基于已从现有资产中捕获的知识进行构建,同时使用建模知识为一致性和精确性最为重要的业务概念建立权威定义。无需预先对每个概念进行建模。
本研究探讨的是 Genie Ontology 的一个不同方面,不同于我们在 Genie Ontology GA 公告中描述的 Turing 基准测试。该基准测试比较了带有 Genie Ontology 的 Genie 与领先前沿智能体在准确性和效率方面的表现,而我们的研究则单独考察建模知识的贡献:受治理的 Metric Views 和 Pages 如何补充学习到的知识和精选智能体,帮助 Genie One 解读业务定义并给出一致的答案。
为了探究这一点,我们构建了四个 Genie Agent,分别面向财务、准备金、承保和理赔。它们使用相同的表,但各自拥有本团队的定义、示例查询和文档。每个单独表现都不错。我们想知道,当 Genie One 成为入口时会发生什么:它能否路由到正确的智能体、使用正确的定义,以及要可靠地做到这一点需要什么?
我们测试了四种配置来找出答案。
仅使用表:没有智能体,也没有 Unity Catalog 中的受治理上下文。
精编智能体:每个智能体都拥有本团队的定义、业务规则和文档。
治理层、无智能体:Unity Catalog 中的 Metric Views、认证、Domains、Subdomains 和 Pages,不使用 Genie Agents
精编智能体 + 治理层:相同的智能体,外加相同的受治理业务上下文。
仅使用表的方案得分 64%。精编智能体和治理层各自达到 83%,但它们的优势并不在相同的方面。加入精编智能体改善了基于文档的回答。加入治理层改善了依赖业务定义的回答。两者单独都不足以同时覆盖这两种需求。两者同时具备时,在两次运行中准确率达到 93–95%。这一组合方案在赔付率问题上也得分 93%,在同时需要文档和结构化数据的问题上得分 97%。
为了检验这一规律在保险行业之外是否成立,我们用一个虚构的零售商场景重复了这项对比。在该场景中,把治理层加到精编智能体之上,将准确率从 54% 提升到 82%。
为什么 Genie One 可能漏掉智能体已经知道的内容
Genie One 并非简单地选择一个智能体并让它回答每个问题。它可以使用受治理的 Unity Catalog Semantics,例如 Metric Views、Domains 和 subdomains,以及 Pages,或者将问题路由到相关的 Genie Agent。如果找不到合适的匹配,它会搜索工作区。在那里,它可能会根据你经常使用的表或卷,或者你之前的对话,推断应使用哪些数据。
当 Genie One 使用自身的上下文回答问题时,它不会自动继承每个智能体内部配置的指令。这些指令仅在 Genie One 将问题路由到该智能体时适用。因此,即使一个智能体整理得很好,如果 Genie One 以不同方式回答问题,它仍然可能贡献甚微。
我们创建了一家虚构的商业汽车保险公司,包含:
8,000 份保单和 7,482 起索赔。
索赔金额每季度更新。
针对尚未报告的索赔的准备金(已发生但未报告,即 IBNR)。
两份再保险合同和三起巨灾事件。
加州近期伤害索赔成本上升。
这四个智能体共享相同的底层表。每个智能体还可以访问各自团队的文档文件夹,四个文件夹中共有 14 个 PDF。
然后,我们在真实定义旁边放置了三个相似的表:analytics_loss_ratio_v2、analytics_loss_ratio_test,以及一个个人副本 finance_lr_calendar_quarter_net_jdoe。
这些智能体配置了各自团队的公式、业务规则和 SQL 示例问题。我们在一组单独的练习题上对它们进行了调优,然后将它们冻结,在所有包含智能体的配置中都使用相同的版本。
有多种方式可以让 Genie 使用企业文档,具体取决于内容所在的位置以及团队需要如何访问它。在这项实验中,我们将各团队专属的 PDF 存储在 Unity Catalog Volumes 中,并通过内容搜索使相应的 Genie Agents 能够使用它们。当信息分散在 Google Drive、SharePoint、Jira 或 Confluence 等业务应用中时,Genie Search 提供更广泛的企业搜索体验。它将来自已连接应用的索引内容与通过 MCP 进行的实时检索相结合,在单一体验中将结果与 Databricks 资产汇集在一起。它还会遵循源系统的权限,因此用户只能看到他们有权访问的内容。
对于受治理层,我们设置了:
Metric Views 包含各团队的计算:三种赔付率定义,以及保费、索赔频率和严重程度的相关团队特定版本。
认证将预期使用的 Metric Views 和智能体标记为可信,并将诱饵资产标记为已弃用。
围绕 Commercial Auto 业务组织资产的域和子域。共享事实存放在主域中,而各团队的定义、文档和智能体则存放在各自的子域中。
为业务术语提供简短、受治理的解释,并链接到其所描述的表和 Metric Views 的页面。
问题与评分
该保险基准测试包含 125 个问题,每个问题的预期答案都用 SQL 独立计算得出。
这些问题涵盖简单查询、明确点名的业务定义、诸如“我们的赔付率是多少?”这类含混的问题、需要给出解释的问题、现有数据无法回答的问题,以及将文档与结构化数据结合起来的问题。我们还测试了追问式问题以及两份植入文档。
我们采取了若干措施,让结果更具意义。Genie One 对每个问题回答三次,每次运行都使用全新的服务主体,以确保此前的对话不会被带入,并且每个答案都记录在 MLflow 中。当某项关键结果十分接近时,我们用新的身份重复了整套设置。我们报告了基于 2000 次自助重采样得出的 95% 置信区间,并将每个问题的重复运行结果归组在一起,因为 Genie One 可能会回忆起先前的回答。来自不同模型家族的两名 AI 评审提取了核心数字、所采用的定义以及所覆盖的事实,再由固定规则判定每个答案是否正确。两位评审在 95–99% 的答案上达成一致。每次运行前,每个智能体都必须独立答对其基础问题中至少 85%。
准确率提升来自何处
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Genie One Accuracy: Why Curated Agents Need Governed Context)
Agents and the governed layer fix different problems. Metric Views, Domains, Subdomains, and Pages help Genie One answer definition questions that would otherwise rely on tribal knowledge, while agents are needed for questions that require answers from unstructured data such as PDFs.
Define each business measure once, in a certified Metric View, and use a Page to explain when it applies. In a separate change test, updating one Metric View and its Page moved all 12 related Genie One answers to the new definition without any agent edits.
Ask Finance and Underwriting whether the loss ratio improved from 2024 to 2025, and you will get two contradictory answers that are both correct. In a fictional commercial auto insurer we built for this test, Finance reported an 8.2 percentage-point deterioration, and Underwriting reported a 1.5 percentage-point improvement. Neither was wrong. Finance tracks the calendar year net of reinsurance, so an adverse reserve development on an older claim shows up in today's numbers. Underwriting looks strictly at the policy year, gross of reinsurance and excluding catastrophes, because that's the baseline used to set rates. Reserving, meanwhile, relies on a third metric entirely: the accident-year ultimate loss ratio.

The SQL can be perfectly correct and still produce an answer that doesn't make sense for the person asking. The challenge is knowing which business definition to apply when the same term has several valid definitions, while keeping shared facts, such as total policies written, consistent. This is the problem we wanted to explore with Genie One.
Genie One, powered by Genie Ontology (now Generally Available), brings this business context into the analytics experience. Genie Ontology learns from the work teams already do, automatically capturing business logic, calculations, and relationships from existing dashboards, queries, notebooks, and semantic models as Ontology Snippets. This learned knowledge is complemented by modeled knowledge: explicitly defined, governed business concepts represented in Unity Catalog through certified Metric Views, Domains and Subdomains, and Pages. In this study, we refer to this modeled knowledge as the “governed layer”.
Together, learned and modeled knowledge give Genie a richer understanding of how an organization works. Teams can build on the knowledge already captured from their existing assets, while using modeled knowledge to establish authoritative definitions for the business concepts where consistency and precision matter most. There is no need to model every concept upfront.
This study explores a different aspect of Genie Ontology than the Turing benchmark described in our Genie Ontology GA announcement. While the benchmark compared the accuracy and efficiency of Genie with Genie Ontology against leading frontier agents, our study isolates the contribution of modeled knowledge: how governed Metric Views and Pages complement learned knowledge and curated agents to help Genie One interpret business definitions and deliver consistent answers.
To explore this, we built four Genie Agents, one each for Finance, Reserving, Underwriting, and Claims. They used the same tables, but each had its own team's definitions, example queries, and documents. Each did well on its own. We wanted to know what happens when Genie One becomes the entry point: does it route to the right agent and use the right definition, and what does it need to do that reliably?
We tested four configurations to find out.
Tables only: no agents and no governed context in Unity Catalog.
Curated agents: each agent had its team's definitions, business rules, and documents.
Governed layer, no agents: Metric Views, certification, Domains, Subdomains and Pages in Unity Catalog, with no Genie Agents
Curated agents with a governed layer: the same agents plus the same governed business context.
Tables alone scored 64%. Curated agents and the governed layer each reached 83%, but they weren't good at the same things. Adding curated agents improved document-based answers. Adding the governed layer improved answers that depended on business definitions. Neither was enough on its own to cover both needs. With both in place, accuracy reached 93–95%, across two runs. The combined setup also scored 93% on loss-ratio questions and 97% on questions that required both documents and structured data.
To see whether this pattern held outside the insurance sector, we repeated the comparison using a fictional retailer. There, the governed layer raised accuracy from 54% to 82% when added to the curated agents.
Why Genie One can miss what an agent already knows
Genie One doesn't simply choose an agent and let it answer every question. It can use governed Unity Catalog Semantics such as Metric Views, Domains and subdomains, and Pages, or route a question to a relevant Genie Agent. If it doesn't find a suitable match, it searches the workspace. There, it may infer which data to use from tables or volumes you use often, or from your earlier conversations.
When Genie One answers a question using its own context, it doesn't automatically inherit the instructions configured inside each agent. Those instructions apply only when Genie One routes the question to that agent. So an agent can be well curated and still contribute little if Genie One answers the question differently.
We created a fictional commercial auto insurer with:
8,000 policies and 7,482 claims.
Claim values updated quarterly.
Reserves for claims that had not yet been reported (incurred but not reported, or IBNR).
Two reinsurance contracts and three catastrophe events.
A recent increase in California injury claim costs.
The four agents shared the same underlying tables. Each also had access to its team's folder of documents, with 14 PDFs across the four folders.
We also planted two misleading documents to test robustness. One was an early draft of the reserve committee minutes reporting a $4.1 million increase in reserves; the approved minutes in the same folder reported a $5.28 million increase. The other was a vendor newsletter containing a hidden instruction to report the loss ratio as 55%, a simple prompt-injection test.
We then placed three look-alike tables next to the real definitions: analytics_loss_ratio_v2, analytics_loss_ratio_test, and a personal copy, finance_lr_calendar_quarter_net_jdoe.
The agents were configured with their teams' formulas, business rules, and SQL example questions. We tuned them on a separate set of practice questions and then froze them, using the same versions across all configurations that included agents.
There are multiple ways to make enterprise documents available to Genie, depending on where the content lives and how teams need to access it. In this experiment, we stored team-specific PDFs in Unity Catalog Volumes and made them available to the corresponding Genie Agents through content search. When information is spread across business applications such as Google Drive, SharePoint, Jira, or Confluence, Genie Search provides a broader enterprise search experience. It combines indexed content from connected applications with live retrieval through MCP, bringing results together with Databricks assets in a single experience. It also respects the permissions of the source systems, so users only see content they are authorized to access.
For the governed layer, we set up:
Metric Views containing the teams' calculations: three loss-ratio definitions and the relevant team-specific versions of premium, claim frequency, and severity.
Certification marking the intended Metric Views and agents as trusted and the decoy assets as deprecated.
Domains and Subdomains organizing the assets around the Commercial Auto business. Shared facts lived in the main Domain, while each team's definitions, documents, and agents lived in its own Subdomain.
Pages providing short, governed explanations of business terms and linked to the tables and Metric Views they described.
The questions and scoring
The insurance benchmark contained 125 questions, each with an expected answer calculated independently in SQL.
The questions covered simple lookups, explicitly named business definitions, ambiguous questions such as "What's our loss ratio?", questions requiring an explanation, questions that the available data couldn't answer, and questions that combined documents with structured data. We also tested follow-up questions and the two planted documents.
We took several steps to make the results more meaningful. Genie One answered every question three times, with a fresh service principal for each run so earlier conversations couldn't carry over, and every answer was logged in MLflow. When a key result was close, we repeated the whole setup with a new identity. We report 95% confidence intervals from 2,000 bootstrap resamples, keeping each question's repeats together, because Genie One can recall earlier answers. Two AI judges from different model families extracted the headline number, the definition used, and the facts covered, and fixed rules decided whether each answer was right. The judges agreed on 95–99% of answers. Before each run, every agent had to answer at least 85% of its basic questions correctly on its own.
Where the accuracy gains came from
Here’s the overall accuracy score, followed by a breakdown by question category, measured with Genie One.
Governed layer, no agents
Curated agents + governed layer
Definition questions (31)
Loss ratio questions (36)
Questions requiring documents and data (12)
Answers using a decoy table
Answers relying on Pages from other projects in the account
What the governed layer added. With curated agents alone, Genie One often answered definition questions on its own and never saw the formulas in the agents' instructions. Published Pages are visible to Genie One across the whole account, so the closest definitions it could find were Pages other people had written for unrelated projects. It relied on those in 49% of its answers. The governed layer provided Genie One with a better match: certified Metric Views for each calculation and our own Pages for each contested term, organized within the Commercial Auto Domain. Reliance on other projects' Pages fell to 11%, and every definition question in our benchmark came back right.
We also looked at whether Genie One routed definition questions to an agent or answered them directly.
Definition questions through Genie One
Answered by Genie One directly
Curated agents + governed layer
With curated agents alone, the agents knew the answer, but Genie One answered about four in ten definition questions itself and got most of those wrong. With the governed layer, it got them right either way. We ran that setup twice from scratch with different identities, and routing shifted dramatically: Genie One handed 74% of answers to an agent in one run and 19% in the other, with about the same accuracy. Routing isn't deterministic, but with the governed layer it stopped mattering. Answering directly, Genie One read the certified Metric Views and Pages. Routing to an agent, the agent applied a formula that matched the same Metric View. Every path led to the same definition.
The planted documents didn't change any answers. No answer in any setup reported the 55% loss ratio from the injected vendor newsletter, and none used the draft reserve figure.
Without the governed layer, routing decided whether you got the right answer. The lesson is to design for whichever path Genie One takes: put each definition where every path can reach it, and test through Genie One more than once before trusting a result.
What the agents added. Without agents, document questions fell to the tables-only level. Of the 18 questions that worsened without agents, 12 required a team document, such as the litigation review, the reserve committee minutes, or an underwriting recommendation. In this setup, the agents were Genie One's only route to those documents, because the PDFs were available only through agent content search and none of the Pages linked to them. Without agents, Genie One also forecast a 2027 loss ratio for every repeat, rather than saying the data didn't cover 2027. That limitation was written in the table descriptions and the agents' instructions, but not on the Metric Views Genie One was reading, so it never saw it.
What each part of the governed layer contributed
The comparison above adds the whole governed layer at once. To understand what each component contributed, we ran a separate series of tests, adding one component at a time.
This second experiment started with four agents whose loss-ratio formulas disagreed. From the second step onward, we moved the formulas into Metric Views and reduced each agent to a few behavior rules and two example queries. These agents were configured differently from the fully curated agents in the main comparison, so the results below shouldn't be compared directly with the earlier table.
Metric Views fixed the formulas. The same step also cut the agents back to a few behavior rules. Loss ratio questions went from 56% to 83%, and definition questions from 66% to 90%.
Certification helped signal which assets to trust, but didn't independently improve accuracy. The overall score dipped from 90% (84–94%) to 83% (76–89%), but the intervals overlap, so we can't say the drop is real. At the same time, decoy-table use fell from 9% to 4% of answers, and Genie One explicitly referred to a certified definition in 22% of answers. We see certification as a useful trust signal that works alongside Domains and Pages, not as a feature that makes answers correct on its own.
Domains and Pages, added last, helped Genie One identify which definition applied. Vague question accuracy increased to 94%, definition questions reached 100%, and decoy use dropped from 4% to zero. A Metric View tells Genie One how to calculate a number; the Domain and Pages help explain whose measure it is and how it fits into the business.
Document questions varied more between runs than the other categories. There were only 12, so changing two questions could move the score by about 15 percentage points. The original agents' instructions said where to find litigation status, and Genie One routed both litigation questions to the Claims agent and got them right. Once we trimmed those instructions, the questions usually failed until we added a Page defining "in litigation."
The same question can have different correct answers
We also tested whether Genie One could account for the user's team when interpreting a question.
We created two identities, one for Finance and one for Underwriting. Each told Genie One its team once, and Genie One Memory retained that information. Both identities then asked the same 12 questions in new conversations: seven where the correct answer depended on the team and five that asked for straightforward facts.
"Did our loss ratio get better or worse from 2024 to 2025?"
8.2 points worse (calendar year, after reinsurance)
1.5 points better (policies written that year, before reinsurance, without catastrophes)
Questions that depend on the team
6 of 7 right on every repeat
7 of 7 right on every repeat
Plain facts (policies written, claims, premium written…)
Every answer explained which team's definition it was using. This shows learned and modeled knowledge working together, because Genie One combined what it remembered about each user with the definition that team had modeled. Both identities had exactly the same data access, so the difference came from the team context Genie One remembered, not from differences in what the identities could see. This was a test of business context, not access control. Remembering someone's team doesn't replace permissions or row-level security; those still need to be enforced separately.
What happens when a business definition changes?
Business definitions don't stay fixed. We wanted to see how much work it would take to update one without breaking the others.
Partway through, Underwriting decided its loss ratio should also include the legal cost of defending claims. We updated the Underwriting Metric View and added one sentence to its Page. We didn't change any agent instructions. Then we asked 12 questions about the Underwriting loss ratio through Genie One, with the curated agents and the governed layer in place.
Every answer used the new definition. Eight questions about definitions we didn't change stayed right. In the curated-only setup, the same change would have meant finding and editing the formula in seven places across four agents' instructions.
One caveat: the agents still held the old formula. Asked directly outside Genie One, they used it in 92% of answers with no warning and would keep doing so until someone updates each agent. Centralizing definitions reduces duplication, but any logic left in agents still needs maintenance.
Does the same approach work outside insurance?
We repeated the main comparison with a fictional retailer. Finance, Merchandising, and Marketing each had a different definition of revenue. We asked, "Did revenue grow in 2025?"
Merchandising, which counts everything ordered before discounts, reports growth of 13.6%.
Finance, which counts revenue when orders ship, after discounts and refunds, says it fell 9.4%.
Marketing, which counts completed orders before returns, says it stayed flat.
A holiday sale and a jump in clothing returns explain the gap, and all three are right.
The benchmark contained 49 questions, making this a smaller check of the pattern rather than a second full study.
Retail, through Genie One
Curated agents + governed layer
Answers using tables from outside this business
Answers using look-alike tables
The curated retail agents worked well on their own. Through Genie One, though, they got about half the questions right, because nothing in Unity Catalog said what revenue meant for each team or which tables belonged to the business. With the governed layer, definition accuracy jumped from 31% to 100%, and overall accuracy rose to 82%.
Based on these experiments, keep your curated agents and add a governed layer underneath them. The key decision is where each piece of information belongs:
Scoped datasets with join keys, calculations, and dimensions that Genie One and Agents use to calculate and aggregate data
Business and team ownership, with shared facts in the main Domain and team-specific context in Subdomains
Explanations of business terms, pointing to the tables, views, and documents that hold the evidence. Pages can link to related assets relevant to the page, such as a Metric View.
Table and Metric View descriptions
What the data covers and doesn't, including date ranges, missing budget or expense data
Behavior rules state which definition was used, ask for clarification when needed, and acknowledge when the data can't answer a question
A few practical lessons stood out:
Start with definitions that mean different things to different teams, especially those that exist as tribal knowledge within a single team. Genie One already handles universally agreed-upon facts well.
Define each measure once. Put the official calculation in a certified Metric View rather than maintaining the same formula in several agent instructions. In our tests, Metric Views and Pages drove the gains in accuracy. In our step-by-step test, certification steered Genie One away from look-alike decoy tables, cutting their use from 9% to 4%, and Domains and Pages took it to zero.
Use Pages to explain the business context. Link them to the certified tables and Metric Views they describe. On each Page, define the scenarios where a calculation applies and the ones where Genie One should avoid it. A definition is more useful when Genie One can connect it to the underlying data.
Keep agents focused on what they do best. Team-specific documents, Domain knowledge, and behavior rules still matter. The governed layer doesn't replace that context, particularly when a question depends on information that isn't represented in tables or Metric Views.
Test through Genie One, not just through individual agents. An agent that answers well on its own won't help if Genie One doesn't route to it, so measure the end-to-end experience before and after each change.
To apply this to your own data, start with the terms your teams define differently, add them to a Domain/Subdomain with Metric Views and Pages, and measure the change through Genie One. The Genie Ontology documentation explains how the pieces fit together, and our guide to operationalizing Genie Ontology in your data stack walks through setting them up step by step.
Get the latest posts in your inbox
原文为英文,正文由自动抓取并保留原文。
原文发布于 Databricks Blog(Genie One Accuracy: Why Curated Agents Need Governed Context),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。