11-19 更新 实时
首页/ 技术栈/ 正文

分支、同步、服务:Lakebase 黑客松的构建启示

技术栈 来源:Databricks Blog 2026-11-18 03:00 0 阅读 机器翻译
分支、同步、服务:Lakebase 黑客松的构建启示

从智能体在触及生产环境之前先在分支上测试其操作,到一家保险公司下线一个独立的搜索索引,这些构建项目说明了为什么运营数据库正在成为数据平台的一部分,而不再是与之并排部署的独立系统。

Databricks 黑客松如何运作

我们举办这些黑客松的理由很简单:学习一个产品最快的方式就是用它来做点东西。每一场黑客松都以一场赋能培训开场,团队在其中了解这款产品以及它在路线图中的发展方向。随后便进入开发阶段。团队合作大约一周并提交项目,最出色的方案可以获得奖品。

这一次,我们把黑客松开放给了比以往范围广得多的客户群体,而处于聚光灯下的产品是 Databricks Lakebase。

为什么 Lakebase 非常适合做黑客松

Lakebase 是全托管的 Postgres,因此任何写过 SQL 查询或用过 ORM 的开发者都能立刻上手开发。它的不同之处在于底层架构。存储与计算分离,数据库与你的 lakehouse 数据放在一起,并处于同一套 Unity Catalog 治理之下。这带来了三点典型的 Postgres 数据库难以做到的能力:

分支。Lakebase 的分支就像数据库的 Git 分支。借助写时复制存储,新分支可以瞬间创建,因此你只需为发生变化的数据付费,而且创建分支对生产环境没有性能影响。

同步。同步表会持续把 Unity Catalog 表以只读表的形式复制到 Postgres,因此应用可以以低延迟查询 lakehouse 数据。这通常被称为反向 ETL。在另一个方向上,Lakehouse Sync 利用变更数据捕获把 Postgres 表复制回 Unity Catalog 的 Delta 表。

提供服务。Lakebase 可以保存应用状态,借助 pgvector、PostGIS 和 pg_trgm 等扩展为搜索提供支持,并充当 Databricks Online Feature Stores 背后的在线存储,用于实时机器学习。

这些对应我们开展的两条路径。路径 1 聚焦于分支和构建应用。路径 2 聚焦于同步和服务。在两者底层,Unity Catalog 管控谁能看到什么,无论数据是在 notebook 中被查询,还是被生产应用读取。

路径 1:使用 Lakebase 分支构建应用

适用对象:应用开发者。这些工程师构建运行在运营数据之上的应用、API 和智能体,他们通常不得不在缓慢的共享预发布数据库与为每位开发者准备昂贵副本之间做选择。分支消除了这种取舍。每个 Lakebase 项目都从一个生产分支开始,你可以从它创建子分支,用于开发、测试或实验。由于分支采用写时复制,大型数据库的分支可在数秒内就绪,且只有变更会占用新存储。这条路径中的团队使用 Genie Code 构建应用,并将分支视为生产环境的廉价、一次性副本。

Lyft 着手解决一个随着 AI 智能体开始对生产数据执行操作而日益突出的问题:智能体在不知道以触发器形式存在于数据库内部的业务规则的情况下,就发起退款、取消预订或关闭工单。他们的项目 Jevons Search 为智能体提供一个 Lakebase 分支作为操作空间。对于每一批支持工单,智能体会在约六秒内 fork 出生产环境的写时复制分支,在其上执行操作,读回数据库实际产生的结果,并据此调整。该项目不用大型 LLM 来做规划,而是用一个基于 gte-large 嵌入构建的轻量级分类器——在 MLflow 中跟踪、在 Model Serving 上提供服务——把每张工单归入八个策略类别之一,运行时完全不消耗 LLM Token。在基于 Wanderbricks 示例数据构建的合成数据集上,预先规划在策略允许范围内完全解决了 11 个支持积压中的 2 个;在分支上测试则解决了 11 个中的 10 个。团队还验证了项目名称背后的理念:Lakebase 的 fork 速度足够快,同一分钟内能跑完的分支数量超过原来的四倍,但分支更多并没有带来更多通过。精确的决策加上来自数据库的真实反馈,胜过蛮力。

CIBC 构建了 Branchflight,一个为每个拉取请求提供专属预览环境的平台。当拉取请求打开时,Branchflight 会在不到五秒内从经过 PII 脱敏的模板分支创建一个零拷贝的 Lakebase 分支,并部署一个连接到该分支的预览版 Databricks App。无人使用时,这些环境会缩容到零,在设定时间后自动过期,并在变更合并前通过模式差异检查。该平台自身的状态、审计日志和成本指标也存放在 Lakebase 中。

Arbital Health 提出了一个所有要把 text-to-SQL 交付给用户使用的团队都该思考的问题:schema 本身对准确率的影响有多大?他们的基准测试 7NF 保持数据固定不变、只改 schema,每种布局都作为独立的 Lakebase 分支运行。模型只按是否返回正确的行来评分,因此团队可以为客户数据库挑选最佳 schema,或改进自己的 text-to-SQL 模型。

分支还出现在一些意想不到的地方。

在黑客松参赛作品中,OneTrust 把 Lakebase 分支引入网络安全领域,将防火墙规则当作代码来对待。为了在不危及线上基础设施的前提下评估拟议的变更,他们尝试让一个 Go 应用启动一个带独立算力的写时复制 Lakebase 分支,并连接一个测试实例来运行递归可达性查询和策略检查。在测试中,会扩大攻击面的试运行被立即丢弃,而验证通过的规则则直接合并到生产环境。

CIBC 的第二支团队打造了 BranchCut,一个 3D 电影化工作空间,把 Git 式的分支与合并带到创意工作中。创作者可以从场景中的任意节点分叉,尝试不同的镜头运动或灯光,并排比较各个版本,再把最佳版本合并回去,Lakebase 则负责跟踪项目、分支、场景状态和合并历史。

Reach Mobile 打造了 StreetCart,帮助购物者按价格、距离和库存比较附近的商店,而不必一家一家地走。Lakebase 是该应用存储商品、门店和库存的业务数据库,Lakebase 搜索则为商品发现提供支持。团队用分支在修改生产 schema 之前测试新的商品类目属性。

Markel 着手改造大多数财产和意外险保险公司都在运行的一种系统:一个独立的搜索索引,通常是 Elasticsearch,之所以保留它,只是为了让理赔、核保和呼叫中心团队能够快速读取。这第二份副本有自己的管道、自己的模式漂移,以及自己的治理缺口。Markel 的应用通过 Lakebase,从用于分析的同一批受治理 Gold 表中满足这些读取需求。理赔分析师按保单号、姓名、地址或电话搜索保单;呼叫中心坐席通过电话号码在保单、理赔和经纪人数据中识别来电者;核保人员在一个地方查看某经纪人的损失记录。新理赔到达时即进行风险评分。

第 2 部分:将 Lakebase 连接到湖仓

在 Lakebase 上构建只是故事的一半。另一半是当操作型数据库和湖仓不再是彼此分离的系统时会发生什么。

面向人群:数据和 ML 工程师。这些人构建管道和模型,并且通常必须构建一条单独的路径,才能把这些数据送入需要在毫秒级内获取它的应用和服务。

这部分旨在弥合这一差距。Synced tables 通过托管的 Spark Declarative Pipelines 将经过整理的 Unity Catalog 数据移入 Lakebase Postgres,使应用可以将其与自己的表连接。Lakebase Change Data Feed 则反向而行,将 Postgres 中所做的更改以流式传输回 Delta 用于分析。并且由于 Databricks Online Feature Stores 运行在 Lakebase 上,在湖仓中定义的特征可以实时提供给模型。

Data Axle 为度假租赁房东打造了 PriceWise,这些房东设定一个固定的每晚价格,而全年需求变化剧烈。在超过 18,000 个房源中,团队估计房东错失了约 64% 的潜在每晚收入。PriceWise 结合了混合检索(pgvector 嵌入加 BM25,用 Reciprocal Rank Fusion 融合)、拆分为四个需求驱动因素的定价建议(包括对 20 英里内房源进行 PostGIS 比较),以及一个嵌入式 Genie 智能体,它能回答诸如头部竞争对手拥有哪些设施之类的问题。每一次 Genie 对话都作为智能体记忆保存在 Lakebase 中,这样房东可以从中断处继续。搜索、定价、智能体记忆和特征服务都运行在同一个 Lakebase 数据库上,并通过同步表保持最新,定价响应最快只需 4 毫秒,房东的决策也会写回 Lakebase。

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Branch, sync, serve: What our Lakebase Hackathon taught us about building on Lakebase)

From AI agents that test their actions on a branch before touching production to an insurer retiring a separate search index, the builds show why the operational database is becoming part of the data platform instead of a system that sits next to it.

How Databricks hackathons work

We run these hackathons for a simple reason: the fastest way to learn a product is to build something with it. Each one starts with an enablement session where teams learn the product and where it’s headed on the roadmap. Then the building starts. Teams work together for about a week and submit a project, and the strongest solutions earn prizes.

This time, we opened the hackathon to a much wider group of customers than ever before, and the product in the spotlight was Databricks Lakebase.

Why Lakebase made for a great hackathon

Lakebase is fully managed Postgres, so any developer who has written a SQL query or used an ORM can start building right away. What makes it different is what sits underneath. Storage is separated from compute, and the database lives next to your lakehouse data under the same Unity Catalog governance. That opens up three things a typical Postgres database can’t do as easily:

Branch it. Lakebase branches work like Git branches for your database. A new branch is created instantly using copy-on-write storage, so you only pay for the data that changes, and creating one has no performance impact on production.

Sync it. Synced tables continuously copy Unity Catalog tables into Postgres as read-only tables, so applications can query lakehouse data with low latency. This is commonly called reverse ETL. In the other direction, Lakehouse Sync replicates Postgres tables back into Unity Catalog Delta tables using change data capture.

Serve it. Lakebase can hold application state, power search with extensions like pgvector, PostGIS, and pg_trgm, and act as the online store behind Databricks Online Feature Stores for real-time ML.

Those map to the two tracks we ran. Track 1 focused on branching and building applications. Track 2 focused on syncing and serving. Underneath both, Unity Catalog governs who can see what, whether the data is being queried in a notebook or read by a production app.

Track 1: Build applications with Lakebase branching

Who it’s for: application developers. These are the engineers who build the apps, APIs, and agents that run on operational data, and who have usually had to choose between a slow, shared staging database and an expensive copy for every developer. Branching removes that tradeoff. Every Lakebase project starts with a production branch, and you can create child branches from it for development, testing, or experiments. Because branches use copy-on-write, a branch of a large database is ready in seconds, and only the changes take up new storage. Teams in this track used Genie Code to build their apps and treated branches as cheap, disposable copies of production.

Jevons Search

Lyft took on a problem that’s growing as AI agents start acting on production data: agents that issue refunds, cancel bookings, or close tickets without knowing the business rules that live inside the database as triggers. Their project, Jevons Search, gives the agent a Lakebase branch to work on. For each batch of support tickets, the agent forks a copy-on-write branch of production in about six seconds, applies its actions, reads back what the database actually did, and adjusts. Instead of using a large LLM to plan, a lightweight classifier built on gte-large embeddings, tracked in MLflow, and served on Model Serving sorts each ticket into one of eight policy categories, with no LLM tokens used at runtime. On a synthetic dataset built from the Wanderbricks sample data, planning up front fully resolved 2 of 11 support backlogs within policy. Testing on branches resolved 10 of 11. The team also tested the idea behind the project’s name: Lakebase forked fast enough to score more than four times as many branches in the same minute, but more branches didn’t lead to more passes. Precise decisions plus real feedback from the database beat brute force.

CIBC built Branchflight, a platform that gives every pull request its own preview environment. When a pull request opens, Branchflight creates a zero-copy Lakebase branch from a PII-masked template branch in under five seconds and deploys a preview Databricks App connected to it. Environments scale to zero when no one is using them, expire automatically after a set time, and pass through a schema-diff check before changes merge. The platform’s own state, audit logs, and cost metrics live in Lakebase too.

Arbital Health asked a question every team putting text-to-SQL in front of users should think about: how much does the schema itself affect accuracy? Their benchmark, 7NF, keeps the data fixed and changes the schema, with each layout running as its own Lakebase branch. Models are graded only on whether they return the correct rows, so teams can pick the best schema for a customer database or improve their own text-to-SQL models.

Branching also showed up in some unexpected places.

For their hackathon submission, OneTrust brought Lakebase branching to network security by treating firewall rules like code. To evaluate proposed changes without risk to live infrastructure, they experimented with having a Go app spin up a copy-on-write Lakebase branch with dedicated compute and connect a test instance to run recursive reachability queries and policy checks. In testing, trial runs that expanded the attack surface were discarded instantly, while validated rules merged directly into production.

A second CIBC team built BranchCut, a 3D cinematic workspace that brings Git-style branching and merging to creative work. Creators branch from any point in a scene, try different camera moves or lighting, compare versions side by side, and merge the best one back, with Lakebase tracking projects, branches, scene state, and merge history.

Reach Mobile built StreetCart, which helps shoppers compare nearby stores on price, distance, and stock instead of walking store to store. Lakebase is the app’s operational database for products, stores, and inventory, and Lakebase search powers product discovery. The team used branching to test new product category attributes before changing the production schema.

Markel went after a system most property and casualty insurers run: a separate search index, often Elasticsearch, kept only so claims, underwriting, and call center teams get fast reads. That second copy has its own pipeline, its own schema drift, and its own governance gaps. Markel’s app serves those reads from the same governed gold tables used for analytics, through Lakebase. Claims analysts search for policies by number, name, address, or phone; call center agents identify callers by phone number across policy, claims, and broker data; and underwriters see loss runs for a broker in one place. New claims are scored for risk as they arrive.

Track 2: Connect Lakebase to the lakehouse

Building on Lakebase is only half the story. The other half is what happens when the operational database and the lakehouse stop being separate systems.

Who it’s for: data and ML engineers. These are the people who build the pipelines and models, and who usually have to build a separate path to get that data into the apps and services that need it in milliseconds.

This track was about closing that gap. Synced tables move curated Unity Catalog data into Lakebase Postgres through managed Spark Declarative Pipelines, so apps can join it with their own tables. Lakebase Change Data Feed goes the other way and streams changes made in Postgres back to Delta for analytics. And because Databricks Online Feature Stores run on Lakebase, features defined in the lakehouse can be served to models in real time.

Data Axle built PriceWise for vacation rental hosts who set one flat nightly price while demand changes dramatically across the year. Across more than 18,000 listings, the team estimated hosts leave about 64% of potential nightly revenue on the table. PriceWise combines hybrid search (pgvector embeddings plus BM25, merged with Reciprocal Rank Fusion), pricing suggestions broken down into four demand drivers including PostGIS comparisons of listings within 20 miles, and an embedded Genie agent that answers questions like which amenities top competitors have. Every Genie conversation is saved in Lakebase as agent memory, so hosts can pick up where they left off. Search, pricing, agent memory, and feature serving all run from one Lakebase database kept current through synced tables, with pricing responses in as little as 4 milliseconds, and host decisions written back to Lakebase.

A team from a transportation company built a real-time flight risk app that turns streaming data into a live operational view. They synced flight, airport, and weather alert tables from Unity Catalog into Lakebase to power a live map in a Databricks App, showing how synced tables let the same pipelines that feed analytics feed an operational app.

Another team from an insurance company focused on a gap that comes after a policy is issued. Risk scoring usually happens before a policy is bound, but no one predicts what the inspector will find afterward. Their app scores how likely each post-bind inspection is to lead to carrier action and ranks the inspection queue so the riskiest policies are seen first, with plain-language reasons. Features are defined once as Databricks Feature Views and served from a Lakebase online store, so the same definitions power training and real-time scoring.

Sobeys built a conversational shopping assistant with three agents. A recipe agent finds recipes with Vector Search, a product agent matches ingredients to store products using fuzzy Postgres trigram search, and a health agent checks every recipe against the family’s allergy profiles. Ask for Thai recipes that are safe for a peanut allergy, and the app goes from recipe to ingredients to cart in one conversation.

Read the two tracks back to back and you get a working tour of Lakebase. A developer gives an AI agent a branch instead of production. A platform team gives every pull request its own database in seconds. A data engineer syncs curated tables into Postgres to power a live app. An ML engineer defines features once and serves them in real time.

What ties all of it together is that Lakebase doesn’t sit apart from the data platform. The same Unity Catalog governance applies to the lakehouse tables and the Postgres tables that serve them, and the same data that powers a dashboard can power an app, an agent, or a model. Several teams replaced a separate system, like a search index, a staging database, or a hand-built sync job, with something Lakebase handles directly.

Hats off to all eleven teams for building something real. A few resources to get started with Lakebase:

What is Databricks Lakebase?

Lakebase is fully managed Postgres built for the lakehouse. Storage is separated from compute, and the database lives next to your lakehouse data under the same Unity Catalog governance, so any developer who has written a SQL query or used an ORM can start building right away.

How does database branching work in Lakebase?

Lakebase branches work like Git branches for your database. Every project starts with a production branch, and new child branches are created instantly using copy-on-write storage, so you only pay for the data that changes and creating a branch has no performance impact on production.

What are Lakebase synced tables?

Synced tables continuously copy Unity Catalog tables into Postgres as read-only tables, so applications can query lakehouse data with low latency. This is commonly called reverse ETL. In the other direction, Lakehouse Sync replicates Postgres tables back into Unity Catalog Delta tables using change data capture.

Can Lakebase power search and real-time ML?

Yes. Lakebase supports Postgres extensions like pgvector, PostGIS, and pg_trgm for search, and it acts as the online store behind Databricks Online Feature Stores, so features defined in the lakehouse can be served to models in real time.

How do Databricks customer hackathons work?

Each hackathon starts with an enablement session on the product and its roadmap. Teams then build together for about a week and submit a project, and the strongest solutions earn prizes. This Lakebase hackathon had two tracks and eleven customer projects.

New to Lakebase? Start with the Lakebase product page and the docs on branches.

Want to bring lakehouse data into your app? Read about synced tables and Lakehouse Sync.

Building search or real-time ML? See the supported Postgres extensions and Lakebase-powered Online Feature Stores.

Ready to try it? Start a free trial and create your first Lakebase project.

Get the latest posts in your inbox

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 Databricks Blog(Branch, sync, serve: What our Lakebase Hackathon taught us about building on Lakebase),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗