11-19 更新 实时
首页/ 技术栈/ 正文

Lakehouse//RT:扩展实时分析,而非账单

技术栈 来源:Databricks Blog 2026-11-13 00:53 0 阅读 机器翻译
Lakehouse//RT:扩展实时分析,而非账单

AI 智能体几乎能在一夜之间让实时查询量成倍增长,团队想知道账单是否也会随之成倍上涨。

Lakehouse//RT 的设计初衷就是承接这类需求,而不会带来成比例的成本上升

选择 Lakehouse//RT 而非独立实时服务层的客户反馈称,基础设施成本降低了 6 到 10 倍

AI 智能体正在挑战数据仓库的极限。人类分析师一下午可能只跑几十条查询;而智能体可以全天候每秒运行数百条查询。随着面向客户的毫秒级刷新仪表盘、内嵌分析的应用等用例需求不断增长,实时工作负载正迅速成为企业使用数据的标准组成部分。

大多数数据仓库的设计面向的是人类规模的查询量,而不是智能体所产生的持续并发。而且即便数据仓库能实现低延迟,账单也会随需求同步增长:查询量增加 10 倍,成本很可能也增加 10 倍。

我们打造 Lakehouse//RT,不仅是为了直接在湖仓上支撑实时工作负载,也是为了在查询量增长时,不让成本按同样的速度上涨。

为什么在 Lakehouse//RT 上做实时分析具备成本效率?

大多数数据仓库的容量随需求同步扩展:查询更多、算力更多、成本更高。这对许多分析工作负载行之有效,因为这类场景的查询量与人数的多少直接相关。实时工作负载则不同。当 AI 智能体和数千名并发用户都需要在毫秒内得到答案时,查询量的增长速度可能远超人员规模,成本也随之上涨。

实时工作负载往往还呈现尖峰特征。通常的处理方式是过度配置:以大批量增加容量并让其持续运行,以便在峰值时延迟仍能保持稳定。结果就是,一天中大部分时间你都在为闲置算力付费。

Lakehouse//RT 颠覆了这一等式。它不是在需求增长时同步放大成本,而是放大效率。其引擎的架构设计目标是在相同资源上处理多得多的查询,因此当需求翻倍(甚至增长十倍)时,你的基础设施能够跟上,而支出不会成比例激增。

Lakehouse//RT 通过在四个方面采取不同做法来实现这一点:

高并发支持,让更多查询共享同一批资源

更快的执行,让每个查询占用算力的时间更短

以小增量自动扩缩容,让容量紧密跟随需求,而不是以整个集群为步长跳跃式调整

用一套系统取代多套系统,这样你就不必再为重复的基础设施以及保持系统同步的工作付费

企业无法靠花钱跑赢需求曲线。唯一可持续的路径,是采用一套在每单位算力上做更多事的系统。

1. 更高并发可让每个数据仓库服务更多查询

并发通常是实时成本增长最快的地方。传统数据仓库一次只能运行有限数量的查询;其余的查询要么排队等待,要么触发扩容。要服务数百个智能体,就意味着仅为了保持响应时间稳定就得运行多个集群。即使这些查询大多是不需要强大算力的小型查找,情况依然如此。

Lakehouse//RT 旨在让每个数据仓库并行运行远多于此的查询。在我们的发布博客中,我们分享过,我们能在每秒 12,000 次查询下实现低于 100 毫秒的延迟,而且我们只会在此基础上继续提升。

每个查询也能更快完成,因此容量能更早释放出来处理下一个查询。同样的峰值负载只需少得多的算力即可承载,而智能体仍能在毫秒级获得答案。

客户正在真实工作负载中体验到这一点。Aurasell 从几个每秒 200 次查询的集群迁移到一个小型 Lakehouse//RT 数据仓库,持续支撑超过 250 QPS,查询执行速度提升 4-5 倍,同时成本降低 10 倍。

继续阅读,了解更多客户成功案例!

2. 更快的查询消耗更少算力

假设你雇了一名承包商。一个人每小时收费 20 美元,需要 10 小时完成。另一个人每小时收费 22 美元,五小时就完成。前者花费 200 美元,后者花费 110 美元。你为完成的工作付费,而不是为小时数付费。

算力也是同样的逻辑:你的账单等于单价乘以运行时长。Lakehouse//RT 的设计目标就是快速完成实时查询,包括执行之外的各个环节,比如规划、路由和返回结果。每个查询占用算力的时间都更短。每天数百万次智能体查询累积起来,省下的就很可观。

3. 需要更多容量时,你只需增加所需的那部分

随着流量增长,每个数据仓库最终都需要更多容量。关键在于每次扩容的步长有多大。

传统数据仓库通常通过复制出完整的自身副本来扩容。在一个 Large 仓库上,就是再启动一个完整的 Large 仓库。集群级扩容对稳定的分析类工作负载效果不错,但实时需求的波动更剧烈。当一次突发流量只需要一个集群的一小部分时,增加整个集群就意味着为突发结束前用不上的容量付费。

Lakehouse//RT 可以灵活增加容量,甚至可以一次只加一个节点,无论你选择哪种规格。实时需求很少以整个集群的规模成块出现。智能体流量的一次激增可能只需要多 10% 的容量,而不是 100%,Lakehouse//RT 就能按这个幅度扩容,而不必翻倍。

以小步长扩容又不损害延迟,说起来容易做起来难。这正是 Lakehouse//RT 的设计初衷,它让容量与需求同步增长,而不是超前于需求。

4. 一套系统消除重复的基础设施

过去,实时意味着要在数据仓库之外再外挂一个独立的服务层,比如 Apache Druid、Apache Pinot 或某个缓存。这意味着常开的基础设施、数据的第二份副本、同步管道,以及负责让这一切保持一致的工程师。

Lakehouse//RT 直接查询受治理的湖仓数据,因此随着数据量增长,无需再并行扩展第二套技术栈。

客户的实际体验

Aurasell:成本效率提升 10 倍,查询速度最高提升 5 倍

Aurasell 的 AI 原生市场推广平台运行着各类智能体,全天候负责挖掘潜在客户、管理销售管道并辅导销售人员。这些智能体式工作流依赖高查询吞吐,Aurasell 希望获得实时性能,而无需将数据迁移或索引到外部系统。借助 Lakehouse//RT,该团队只需一个小型集群,就能直接在 Databricks 中承载这一工作负载。

“在 Aurasell,高查询吞吐对我们的智能体式工作流至关重要。在采用 Lakehouse//RT 之前,要达到每秒 200 次查询,我们需要启动多个独立集群,这增加了复杂性。借助 Lakehouse//RT,一个小型集群就能轻松维持超过 250 QPS,同时查询执行速度快 4-5 倍。除了通过整合集群带来基础设施和成本效率 10 倍的提升外,最大的收益是架构上的简化;我们无需将数据迁移或索引到外部系统,就能直接在 Databricks 中获得所需的实时性能。” – Srinivas Bandi,Aurasell CTO

VisionHeight:成本降低 90% 以上,三天内上线生产

VisionHeight 是一个攻击前情报平台,可实时绘制公共互联网上的对手基础设施。其此前的实时层运行在 Apache Pinot 上,使用 7×24 全天候运行的常开 EC2 集群。切换到 Lakehouse//RT 的无服务器模式后,VisionHeight 的成本降低了 90% 以上。在负载最重的查询上,Lakehouse//RT 实现了亚秒级响应时间,比 Pinot 快近 3 倍。整个迁移仅耗时三天。

“用上 Lakehouse//RT 之后,速度非常快。不用把数据抽取出来,不用维护基础设施,成本还降低了约 90%。现在这一切在 Databricks 上都能直接跑通。”—— Nitsan Daniel,VisionHeight 首席技术官兼联合创始人

nOps:基础设施成本降低 6 倍,消除 40TB 重复数据

nOps 为数百家客户管理着每年超过 50 亿美元的云支出,支撑企业实时追踪资金具体流向的实时仪表盘。其实时查询层此前运行在 Apache Druid 上,需要一套独立的全天候常开集群,以及从湖仓同步而来的 40TB 重复数据。迁移到 Lakehouse//RT 之后,nOps 将基础设施成本削减了 6 倍以上,同时彻底消除了重复数据。由于 Lakehouse//RT 直接在数据所在之处进行查询,此次迁移无需复制数据、无需新建数据摄取管道,也无需新增基础设施。

“谁都知道,把超过 380TB 的数据在多个平台之间搬来搬去是场噩梦,而且成本高昂。有了 Lakehouse//RT,我们在完全不改动底层数据的情况下就获得了实时能力。”—— Jordan Stein,nOps 工程总监

AI 与用户需求的浪潮不是即将到来,而是已经到来。查询量将增长数个数量级。问题不在于你是否负担得起实时性能,而在于你是否负担得起没有一个为规模化效率而构建的系统。

Lakehouse//RT 为你带来:

在受治理的湖仓数据上,以低于 100ms 的延迟和数千并发 QPS 满足实时分析需求的性能

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Lakehouse//RT: Scale real-time analytics, not your bill)

AI agents can multiply real-time query volume almost overnight, and teams want to know whether their bill will multiply with it.

Lakehouse//RT is intentionally built to absorb that demand without a proportional cost increase

Customers that chose Lakehouse//RT over separate real-time serving layers report 6x to 10x lower infrastructure costs

AI agents are pushing the limits of data warehouses. A human analyst might run a few dozen queries in an afternoon. An agent can run hundreds per second, around the clock. And with growing demand for use cases like customer-facing dashboards that refresh in milliseconds and applications with embedded analytics, real-time workloads are quickly becoming a standard part of how businesses use data.

Most data warehouses were designed for human-scale query volume, not the sustained concurrency agents generate. And even when a warehouse can deliver low latency, the bill grows as fast as demand: 10x the queries could easily mean 10x the cost.

We built Lakehouse//RT not only to serve real-time workloads directly on the lakehouse, but to do so without growing your costs at the same rate as your queries. 

Why is real-time analytics cost efficient on Lakehouse//RT?

Most data warehouses scale capacity in step with demand: more queries, more compute, more cost. That works well for many analytics workloads, where query volume tracks the number of people asking questions. Real-time workloads are different. When AI agents and thousands of concurrent users all need answers in milliseconds, query volume can grow far faster than headcount, and cost grows with it.

Real-time workloads are also often spiky. The usual way to handle that is to overprovision: add capacity in large blocks and keep it running so latency holds up at peak. You end up paying for idle compute most of the day.

Lakehouse//RT flips the equation. Instead of scaling cost with demand, it scales efficiency. Its engine is architected to serve dramatically more queries on the same resources, so when demand doubles (or increases tenfold), your infrastructure keeps pace without a proportional spike in spend. 

Animation of a demand curve. Agent demand grows at a steep angle, with traditional warehouse costs following the same curve. Lakehouse//RT cost is a much flatter curve, with the gap indicating savings.

Lakehouse//RT achieves this by taking a different approach in four areas:

High concurrency support, so more queries share the same resources

Faster execution, so each query uses compute for less time

Autoscaling in small increments, so capacity follows demand closely instead of jumping in full-cluster steps

One system instead of several, so you stop paying for duplicate infrastructure and the work of keeping the system in sync

Organizations will not be able to outspend the demand curve. The only sustainable path is a system that does more per unit of compute.

1. Higher concurrency serves more queries per warehouse

Concurrency is where real-time costs usually grow fastest. Traditional warehouses can only run so many queries at once; the rest wait in line or trigger more capacity. Serving hundreds of agents then means running several clusters just to keep response times steady. That's true even when most of those queries are small lookups that don't need the horsepower.

Lakehouse//RT is designed to run far more queries side by side on each warehouse. In our launch blog we shared that we can deliver sub-100 millisecond latency at 12,000 queries per second, and we’re only getting better from here.

Each query also finishes quickly, so capacity frees up sooner for the next one. The same peak fits on far less compute, and the agents still get answers in milliseconds.

Customers are experiencing this on real-world workloads. Aurasell moved from several clusters at 200 queries per second to one small Lakehouse//RT warehouse sustaining more than 250 QPS, executing queries 4-5x faster while reducing costs by 10x.

Keep reading for more customer success stories!

2. Faster queries use less compute

Say you hire a contractor. One charges $20 an hour and takes 10 hours. Another charges $22 an hour and finishes in five. The first costs $200 and the second costs $110. You pay for the finished job, not the hour.

Compute works the same way: your bill is rate times runtime. Lakehouse//RT is built to finish real-time queries fast, including the steps around execution like planning, routing, and returning results. Each query holds compute for less time. Across millions of agent queries a day, that adds up.

3. When you need more capacity, you add only what you need

Every warehouse eventually needs more capacity as traffic grows. What matters is the size of each step.

Traditional warehouses typically scale by spinning up whole copies of themselves. On a Large warehouse, that's another Large warehouse spinning up. While cluster-level scaling works well for steady analytics workloads, real-time demand is spikier. When a burst needs only a fraction of a cluster, adding a whole cluster means paying for capacity you don't use until the burst passes.

Lakehouse//RT can flexibly add capacity, even one node at a time, whatever t-shirt size you select. Real-time demand rarely arrives in cluster-sized blocks. A spike in agent traffic might need 10% more capacity, not 100%, and Lakehouse//RT can add a step that size instead of doubling.

Scaling in small steps without hurting latency is easier said than done. It's what Lakehouse//RT was designed for, and it lets capacity grow in step with demand instead of ahead of it.

4. One system removes duplicate infrastructure

Real-time used to mean bolting a separate serving layer onto your warehouse, such as Apache Druid, Apache Pinot, or a cache. That meant always-on infrastructure, a second copy of your data, sync pipelines, and engineers to keep it all consistent. 

Lakehouse//RT queries governed lakehouse data directly, so as volume grows there's no second stack growing alongside it.

What customers are seeing

Aurasell: 10x better cost efficiency, up to 5x faster queries

Aurasell's AI-native go-to-market platform runs agents that prospect, manage pipeline, and coach reps around the clock. Those agentic workflows depend on high query throughput, and Aurasell wanted real-time performance without moving or indexing data into an external system. With Lakehouse//RT, the team serves that workload directly in Databricks from a single small cluster.

"High query throughput is critical for our agentic workflows at Aurasell. Before Lakehouse//RT, reaching 200 queries per second required us to spin up several separate clusters, which added complexity. With Lakehouse//RT, a single small cluster easily sustained over 250 QPS while executing queries 4-5x faster. Beyond the 10x improvement in infrastructure and cost efficiency by consolidating clusters, the biggest win is architectural simplicity; we get the real-time performance we need directly in Databricks without having to move or index our data into external systems." – Srinivas Bandi, CTO, Aurasell

VisionHeight: 90%+ cost reduction, production in three days

VisionHeight is a pre-attack intelligence platform that maps adversary infrastructure across the public internet in real time. Their previous real-time layer ran on Apache Pinot with always-on EC2 clusters running 24/7. After switching to Lakehouse//RT's serverless model, VisionHeight saw a cost reduction of more than 90%. On their heaviest query, Lakehouse//RT delivered sub-second response times, nearly 3x faster than Pinot. The entire migration took three days.

"Once we got Lakehouse//RT, it was very fast. No extracting data out, no maintaining infrastructure, and about a 90% cost reduction. It all just works in Databricks now." – Nitsan Daniel, CTO and Co-founder, VisionHeight

nOps: 6x lower infrastructure costs, 40TB of duplicated data eliminated

nOps manages more than $5 billion in annual cloud spend for hundreds of customers, powering real-time dashboards where enterprises track exactly where their money goes. Their real-time query layer previously ran on Apache Druid, requiring a separate always-on cluster and 40TB of duplicated data synced from their lakehouse. After migrating to Lakehouse//RT, nOps cut infrastructure costs by more than 6x while eliminating the duplicated data entirely. Because Lakehouse//RT queries data directly where it lives, the migration required no data copying, no new ingestion pipelines, and no new infrastructure.

"Anyone would know that trying to move over 380 terabytes of data around to different platforms is a nightmare, and it's expensive. With Lakehouse//RT, we get real-time capabilities with zero change to the underlying data." – Jordan Stein, Director of Engineering, nOps

The wave of AI and user demand isn't coming; it's here. Query volumes will grow by orders of magnitude. The question isn't whether you can afford real-time performance. It's whether you can afford not to have a system built for efficiency at scale.

Lakehouse//RT gives you:

Performance that matches real-time analytics needs with sub-100ms latency and thousands of concurrent QPS on governed lakehouse data

A bill that grows slower than your query volume because each query uses less compute and capacity is added in small steps

No separate infrastructure to buy, sync, or maintain; customers are seeing 6x to 10x lower total infrastructure costs compared to using a separate serving layer for real-time

Lakehouse//RT is available in beta today. Spin up a new Lakehouse//RT warehouse and start testing your most demanding real-time workloads.

Enable Lakehouse//RT: Learn how to start using Lakehouse//RT

Review pricing: Through January 2027, try Lakehouse//RT at a 30% discount. 

Talk to your account team: They can help you model the cost impact for your specific workloads.

Get the latest posts in your inbox

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 Databricks Blog(Lakehouse//RT: Scale real-time analytics, not your bill),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗