与临床数据对话:解锁 AWS HealthLake

如今,HealthLake 数据可以通过 Unity Catalog 进行治理,在笔记本和仪表板中查询,并借助 Genie One 以自然语言进行探索,从而让技术用户和非技术用户都能获得临床洞察。
许多医疗健康和生命科学组织已标准化采用 AWS HealthLake 来管理临床数据,但要将这些数据扩展到其他平台用于分析和 AI,传统上需要付出更多努力。对于希望将 HealthLake 中的数据与外部分析工具配合使用的公司而言,这一流程需要构建导出管道、复制数据,并在每个位置分别管理治理。这限制了 HealthLake 所管理的 FHIR 数据在其他工具和平台之间的互操作性。
在本博客中,我们介绍一种利用 OpenSharing(A Linux Foundation AI & Data Project)的新解决方案,实现 HealthLake 与 Databricks 之间首创性的集成。该架构允许用户直接从 Databricks 实时查询 HealthLake 数据,将 Databricks 的治理、分析和 AI 能力带给临床数据集。通过与 HealthLake 集成,Databricks 可以帮助组织从 FHIR 数据中挖掘更深入的洞察,并开启更广泛的分析与 AI 用例。此外,同样的临床数据现在还可以通过 Genie One 用自然语言进行探索,让用户无需专门的技术工作流程或数据复制即可获得洞察。
当前临床数据分析面临的挑战
快速医疗互操作性资源(FHIR)标准是一种数据格式,用于在不同应用和照护场景之间交换健康信息。AWS HealthLake 是一项托管医疗数据服务,用于存储和处理 FHIR 格式的数据。医疗机构常使用 HealthLake 对此类 FHIR 数据进行规范化,同时也借助其符合标准的 API 来访问患者病历、观察记录和就诊记录等资源。
HealthLake 将其分析表以 Apache Iceberg 表的形式存储在 HealthLake 服务自有的 Amazon S3 存储桶中。客户的 IAM 角色无法直接读取该存储桶,因此 Unity Catalog 等外部工具没有直接访问底层数据的路径。因此,以往要在外部平台上访问这些数据进行分析,必须先通过导出管道将 FHIR 数据复制出 HealthLake,然后才能进行查询。
尽管许多机构依赖 HealthLake 专为 FHIR 打造的数据存储,但要在此基础上扩展治理、分析和自然语言探索能力却并不那么简单。此次与 Databricks 的新集成,能够从这些临床数据中挖掘出有价值的新洞察。
将 Databricks 接入 HealthLake 数据
如下方架构图所示,我们的新集成使用 OpenSharing 安全地开放 HealthLake 中的数据,无需任何导出管道或数据复制。这样一来,Databricks 便可直接读取 HealthLake 数据,并由 Unity Catalog 作为共享目录进行治理。
图 1. 解决方案架构
目前,HealthLake 提供对其底层 Iceberg 表直接访问的唯一方式,是通过 Lake Formation 生成的临时凭证,而 Unity Catalog 自身无法请求这些凭证。该解决方案以一个自定义的 OpenSharing 服务器填补了这一空白,该服务器以 Databricks App 的形式运行,属于全托管的无服务器部署。每次收到表请求时,服务器都会使用 Lake Formation 凭证,解析该表当前的 Iceberg 快照,对数据文件进行预签名,并以标准 OpenSharing 响应返回。随后 Databricks 计算资源直接从 S3 读取这些文件,因此没有数据流经该服务器。AWS 通过 OIDC 联合身份认证信任该应用自身的 Databricks 身份,从而保护敏感的医疗健康数据。Databricks 像挂载其他任何共享一样,将该共享挂载到 Unity Catalog 中,使数据可用于受治理的分析、notebook、仪表板、AI,以及借助 Genie One 的自然语言探索。
图 2. Unity Catalog 中由 HealthLake 管理的 FHIR 数据,展示于 Unity Catalog Explorer
用 Databricks 扩展 HealthLake 的优势
借助 Unity Catalog 实现受治理的访问
如今 HealthLake 数据已可在 Databricks 中读取,各机构可以将其临床数据直接纳入 Unity Catalog 的治理之下。Unity Catalog 成为访问入口,通过共享目录上的授权、细粒度访问控制(FGAC)、基于属性的访问控制(ABAC)和基于角色的访问控制(RBAC),规定谁有权查看和查询每张表。访问和使用情况也可以像 Unity Catalog 中的任何其他数据资产一样接受审计,使临床数据更易于治理和保障安全。
借助 Genie One 进行自然语言探索
如今,同一份共享数据可以为 Databricks 中的 Genie One 提供支撑,让技术水平较低的用户也能分析 HealthLake 数据。这意味着临床分析师、运营团队或战略负责人现在可以直接探索 HealthLake 数据,无需向数据团队提交工单,也不必等待定制报表。
组织可以基于共享目录创建和整理 Genie Agent:定义核心临床数据表、添加通俗语言的描述,并教它们如何定义和解读 A1C 等临床概念。随后,用户可以提出类似“平均 A1C 按月趋势如何?”的问题,Genie One 会理解上下文、找到经过整理的 Genie Agent、生成 SQL,并返回附带表格和图表的汇总答案,同时遵循 Unity Catalog 的权限控制。过去需要 SQL 技能或定制仪表板才能回答的问题,如今可在数秒内得到答案,而 HealthLake 管理的 FHIR 数据中所蕴含的洞察,也能为更广泛的临床、运营和业务用户所用。
图 3. Genie One 让终端用户洞察 HealthLake 管理的 FHIR 数据
为 HealthLake 客户解锁新机遇
这一集成让 HealthLake 突破了以往可用的分析路径。组织不再需要把数据复制并导出到外部分析工具,也不必仅依靠 Athena 或 Redshift 进行分析。取而代之的是,用户现在可以选择直接在 Databricks 中处理 HealthLake 数据,用于受治理的分析、笔记本、仪表板和 AI,从而从临床数据中释放新的价值与洞察。
对于管理复杂临床数据环境的 Databricks 客户而言,这一潜力尤为显著。借助该解决方案,医疗与生命科学领域的客户现在可以在 HealthLake 中的临床数据之上应用 Databricks 原生能力。有了集成 HealthLake 与 Databricks 的路径,这些客户现在可以按业务所需的速度加速推进,而不再受以往管理复杂数据管道所带来的摩擦困扰。这为 HealthLake 与 Databricks 的共同客户打开了新的用例之门。通过创建与 Databricks 的集成路径,该解决方案可帮助医疗与生命科学组织从其 HealthLake 数据中获取额外价值,加速整个领域的分析与洞察。
与你的临床数据对话:立即使用我们完整的参考实现,开始将 AWS HealthLake 数据连接到 Databricks,这是一个以 Declarative Automation Bundle 形式部署的 OpenSharing 服务器,作为官方 Databricks Industry Solutions 仓库提供。
在收件箱中获取最新文章
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Talk to your clinical data: unlocking AWS HealthLake with OpenSharing and Genie One)
HealthLake data can now be governed through Unity Catalog, queried in notebooks and dashboards, and explored in natural language via Genie One, unlocking clinical insights for technical and less-technical users alike.
Many healthcare and life sciences organizations have standardized on AWS HealthLake to manage clinical data, but extending that data to other platforms for analytics and AI has traditionally required more effort. For companies seeking to use data living in HealthLake with external analytics tooling, this process has required building export pipelines, duplicating data, and managing governance separately in each location. This has limited the interoperability of HealthLake-managed FHIR data across other tools and platforms.
In this blog, we present a new solution leveraging OpenSharing (A Linux Foundation AI & Data Project) to enable a first-of-its-kind integration between HealthLake and Databricks. This architecture allows users to query HealthLake data live from Databricks, bringing Databricks governance, analytics, and AI capabilities to clinical datasets. By integrating with HealthLake, Databricks can enable organizations to unlock deeper insights from FHIR data and open the door to a broader set of analytics and AI use cases. Furthermore, that same clinical data can now be explored through natural language with Genie One, allowing users to surface insights without specialized technical workflows or data duplication.
Current challenges in clinical data analytics
The Fast Healthcare Interoperability Resources (FHIR) standard is a data format which is used to exchange health information across different applications and care settings. AWS HealthLake is a managed healthcare data service for storing and working with data in FHIR format. Healthcare organizations often use HealthLake to normalize this FHIR data, as well as for its standards-compliant API for resources such as patient records, observations, and encounters.
HealthLake stores its analytics tables as Apache Iceberg tables in an Amazon S3 bucket owned by the HealthLake service. Customer IAM roles can’t read that bucket directly, so external tools such as Unity Catalog have no direct path to the underlying data. As a result, accessing this data for analytics on external platforms has previously required export pipelines that copy FHIR data out of HealthLake before it can be queried.
While many organizations rely on HealthLake’s purpose-built FHIR data store, extending it with governance, analytics, and natural language exploration has been less straightforward. This new integration with Databricks unlocks valuable new insights from this clinical data.
Bringing Databricks to HealthLake data
Our new integration, shown in the architecture below, uses OpenSharing to securely expose data in HealthLake without requiring any export pipelines or data duplication. This allows HealthLake data to be read directly by Databricks and governed by Unity Catalog as a shared catalog.
Figure 1. Solution Architecture
Today, the only way HealthLake provides direct access to its underlying Iceberg tables is through temporary credentials generated by Lake Formation, and Unity Catalog cannot request these credentials itself. This solution fills that gap with a custom OpenSharing server that runs as a Databricks App, a fully managed serverless deployment. On every table request, the server uses Lake Formation credentials, resolves the table’s current Iceberg snapshot, pre-signs the data files, and returns them in a standard OpenSharing response. Databricks compute then reads the files directly from S3, so no data passes through the server. AWS trusts the app’s own Databricks identity through OIDC federation, protecting sensitive healthcare data. Databricks mounts the share in Unity Catalog like any other share, making the data available for governed analytics, notebooks, dashboards, AI, and natural language exploration with Genie One.
Figure 2. HealthLake-managed FHIR data in Unity Catalog, shown in Unity Catalog Explorer
Benefits of extending HealthLake with Databricks
Governed access with Unity Catalog
With HealthLake data now readable in Databricks, organizations can bring their clinical data directly under Unity Catalog governance. Unity Catalog becomes the entry point for access, with grants, fine-grained access controls (FGAC), attribute-based access controls (ABAC), and role-based access controls (RBAC) on the shared catalog specifying who is permitted to see and query each table. Access and usage can also be audited like any other data asset in Unity Catalog, making clinical data easier to govern and secure.
Natural language exploration with Genie One
That same shared data can now back Genie One in Databricks, broadening analysis of HealthLake data to less-technical users. This means that clinical analysts, operational teams, or strategy leaders can now explore HealthLake data directly, without filing a ticket with the data team or waiting for a custom report.
Organizations can create and curate Genie Agents over the shared catalog by defining core clinical tables, adding plain-language descriptions, and teaching them how to define and interpret clinical concepts like A1C. A user can then ask a question like “How is average A1C trending by month?” and Genie One understands the context, finds the curated Genie Agent, generates the SQL, and returns the summarized answer with a table and chart, all while respecting Unity Catalog permissions. Questions that previously required SQL skills or custom dashboard creation can now be answered in seconds, and the insights living in HealthLake-managed FHIR data become accessible to a broader set of clinical, operational, and business users.
Figure 3. Genie One giving end users insights into HealthLake-managed FHIR data
Unlocking new opportunities for HealthLake customers
This integration extends HealthLake beyond the analytics paths that have traditionally been available. Organizations no longer need to copy and export data to external analytics tools, or rely solely on Athena or Redshift for analytics. Instead, users now have the option to work with HealthLake data directly in Databricks for governed analytics, notebooks, dashboards, and AI, unlocking new value and insights from clinical data.
The potential is especially significant for Databricks customers managing complex clinical data environments. With this solution, healthcare and life science customers can now apply Databricks-native capabilities on top of their clinical data in HealthLake. With a path to integrate HealthLake and Databricks, these customers can now accelerate at the pace their business needs, without the previous friction of managing complex data pipelines. This opens the door to new use cases for joint HealthLake and Databricks customers. By creating an integration path with Databricks, this solution can help healthcare and life sciences organizations get additional value from their HealthLake data, accelerating analysis and insights across the field.
Talk to your clinical data: get started connecting AWS HealthLake data to Databricks today with our complete reference implementation, an OpenSharing server you deploy as a Declarative Automation Bundle, available as an official Databricks Industry Solutions repo.
Get the latest posts in your inbox
原文为英文,正文由自动抓取并保留原文。
原文发布于 Databricks Blog(Talk to your clinical data: unlocking AWS HealthLake with OpenSharing and Genie One),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。