11-19 更新 实时
首页/ 技术栈/ 正文

破除关于 LTAP 的种种误解

技术栈 来源:Databricks Blog 2026-11-13 05:49 0 阅读 机器翻译
破除关于 LTAP 的种种误解

点查读取 Postgres 堆页面,扫描读取 Parquet,每个分析查询看到的都是一致的快照,包括尚未进入湖中的提交。

逻辑复制通过一条必须持续跟上的流水线重放每一次变更,这使它成为构建湖副本的一种代价高昂的方式。

Reynold Xin 的文章《From monolith to Lakebase to LTAP》解释了 LTAP 是什么,以及 Lakebase 架构如何使其成为可能:一份受治理的运营数据副本,同时服务 Postgres 和分析,无需 CDC 流水线,也无需第二个数据库。

传统数据库将其 WAL 和数据文件保存在一台机器上。Lakebase 将它们从单一机器上移出,放入独立的、可扩展的存储中。

随后的许多评论都是用人们已经熟悉的系统来解读这一说法。有四种解读反复出现:

“LTAP 不过是换个名字的 CDC”

“数据的单一副本只能有一种物理布局,因此要么点查慢,要么扫描慢”

“对象存储上的 Iceberg 或 Delta 表天然就是过时的”

“Postgres 已经提供了抽取路径:逻辑复制”

这些误解都建立在关于 Lakebase 如何存储数据的错误假设之上。我们逐一来看。

误解 1:“LTAP 就是复制”

实际情况是:LTAP 在湖中为每张表保留一份持久化副本。位于其前方的各层持有部分相同的字节,以使该表既持久又快速,但它们都不是第二个数据库。

从外部看,LTAP 可能像一条熟悉的管道。Postgres 写入行,某个进程跟踪日志,变更落入 Iceberg 表,分析侧读取 Iceberg。这条管道就是 CDC。它也是逻辑复制、镜像以及大多数 zero-ETL 产品的结构。因此很容易把 LTAP 归为同一种设计。

要看清它为何并非如此,有必要精确界定什么才算“第二份副本”。一个系统最终在多处持有同一份数据,有两种不同的方式:

复制通过把变更从一个数据库传送到另一个数据库,来保持两个数据库一致。双方各有自己的目录和自己的时间线,任意一方都可以作为事实来源被查询。如果传输停滞,两份副本就会逐渐偏离,没有任何机制强制它们重新一致。

分层(tiering)只保留一个数据库,并在其前面放一组缓存栈。缓存持有部分相同的字节,使读取更快。如果读取未命中缓存,就下探一层,付出一定延迟代价。它从不会创建第二个目录,而且缓存丢失时,表本身不会丢失。

所以测试很简单:这一层能否作为独立的数据库被查询,或者被故障转移过去?如果可以,它就是副本。如果不可以,它就是一层。

复制让两个数据库保持同步。LTAP 只保留一个数据库,把数据湖作为持久层,并在其前面放置缓存

按照这个标准,Postgres 一直都是分层的。WAL、缓冲区缓存和索引所存储的字节都与堆重叠。没有人会把索引称作表的副本,因为它没有自己的目录或时间线。

LTAP 保留了同样的层级结构。它把这一结构搬到对象存储上,并让湖仓引擎读取最底层的持久层。

以下是 Lakebase 中这一层级结构的样子,先看一次写入,再看一次读取:

当写入的 WAL 到达多数 safekeeper 节点时,这次写入才算持久化。随后 pageserver 应用日志,把热页保留在内存和本地磁盘上,并将数据物化到对象存储中。

读取从计算节点开始,该节点维护着缓冲区缓存和本地文件缓存。索引、TOAST 和可见性映射位于同一条热路径上。没有缓存在那里的页面则来自 pageserver。

对象存储本身也是同样的运作方式。当你把一个对象 PUT 到 S3 时,它会被存储多份,通常借助纠删码跨可用区存放。API 仍然只指向一个对象,副本之间也不会运行同步任务。Lakebase 把同样的思路叠加在数据湖前面:日志、缓存、pageserver、对象存储。正是这些额外的文件,让一张表既持久又快速。

CDC 确实会创建第二个数据库。它把逻辑变更流解码到一个外部 schema 中,通常以主键为键,再 upsert 到一张源端永远不会回读的表中。这第二张表可以单独查询,但它可能与源端脱节:类型可能发生漂移,DDL 可能丢失,复制槽可能丢行,而没人选中的表则永远不会出现。结果是两套 catalog。

LTAP 没有需要保持同步的第二张表。也不存在需要选择加入的表清单。Postgres 中存在的表就存在于湖中,而主库对外提供的页只是该存储的缓存。

误区二:“单一副本意味着查询慢或扫描慢”

实际情况是:同一批行的两种无损编码可以服务不同的访问路径——堆页用于点查,Parquet 用于扫描。每个查询读取对自身而言快速的那种布局。

即使这些额外文件构成某种层次结构,布局仍是另一个独立问题。点查在行式布局上高效,扫描在列式布局上高效。如果磁盘上只有一种编码,那么这两种访问模式中必有一种要在 I/O 上付出代价。把这张表称为单一逻辑副本并不能改变这一点。

这个推理是正确的,但它建立在一个假设之上——一份副本就意味着一种编码。只要两种格式能够无损地表示同一批行,这个假设就不成立。

所以问题在于 LTAP 的两种格式是否无损:

主格式是 Postgres 堆页,非常适合点查和更新。

第二种格式是为分析型访问而构建的:Delta 或 Iceberg 之下的 Parquet。值被完整保留,文件携带了足够的物理元数据(堆块、偏移量和 MVCC 版本),因此原始页面可以被重建。Parquet 无法无损表示的值不会被强制转换或丢弃;它们在同一张表的一个溢出字段中传输。

由于没有任何丢失,这份列式副本可以成为湖中的持久化存储,而堆页面则成为它的缓存。这样,每条访问路径都读取适合自己的格式。点查命中计算节点、本地磁盘和 pageserver 上的 8 KB 页面。扫描则通过 OLAP 引擎命中 Parquet。索引仍留在热层中,不会被转码成 Parquet。Postgres 已经是这样工作的:它为同一批行保留两种物理结构以服务于不同的访问路径——一个堆和一个覆盖索引。没有人会把索引当作 customer 的 CDC 副本。SELECT * FROM t WHERE id = 1 和 SELECT count(*) FROM t 是同一张表上的两条访问路径。

点查读取堆页面,扫描读取 Parquet。页面可以从 Parquet 重建,因此两者持有相同的行

有一个更狭义的说法是成立的。来自 Postgres 的列式文件是按堆顺序排列的:行按页号顺序存放,因为每次压缩重写的单位是页。这种布局对于重建页面以及只重写脏页很高效。但作为 WHERE order_date >= … 的聚簇键,它表现很差。如果仓库查询需要这种聚簇方式,可以像 silver 表从任意操作型数据源派生那样把它派生出来。不再需要的是那条以创建湖副本为目的的 CDC 管道。

误区 3:“开放表格式意味着最终一致性”

现实情况是:每条分析型查询读取的都是一个一致的快照。尚未进入数据湖的提交会从 pageserver 读取,因此查询开始前已提交的数据都不会被遗漏。

Iceberg 和 Delta 在对象存储上提供快照隔离,而对象存储的延迟很高。因此很自然地会得出这样的结论:分析总是滞后的,许多小表无法保持数据新鲜,跨表 join 也无法保持一致。这一描述符合将 CDC 数据摄取进数据湖的场景。

但这取决于一个假设:查询只能看到已经落入对象存储中的数据。在 LTAP 中,查询还能看到尚未进入对象存储的页面。这会改变答案,而跟着一条查询走一遍最容易看清这一点。

当一条分析型查询开始时,

它会向 Postgres 询问当前的 LSN。这是一次元数据查询。

它会从对象存储读取几乎全部数据:截至该 LSN 已经物化的所有内容。不与 OLTP 节点共享 CPU 的引擎会执行这次读取。

它会从 pageserver 合并其余部分。在上一次物化之后提交的页面来自 pageserver,并被合并叠加其上。

Postgres 完全不承担批量读取流量——它只返回一个数字,至多再返回那些最新的页面。数据湖并不需要包含最新的一次提交,才能让该提交可见;它需要包含几乎全部内容,外加一种获取上一次物化之后提交的页面的方式。LSN 就是分界点:它告诉引擎合并需要推进到多远。

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Dispelling LTAP myths)

Point lookups read Postgres heap pages and scans read Parquet, and every analytical query sees a consistent snapshot, including commits not yet in the lake.

Logical replication replays every change through a pipeline that must keep up, which makes it a costly way to build the lake copy.

Reynold Xin's post "From monolith to Lakebase to LTAP" explained what LTAP is and how the Lakebase architecture makes it possible: one governed copy of operational data, serving both Postgres and analytics, with no CDC pipeline and no second database.

A traditional database keeps its WAL and data files on one machine. Lakebase moves them off a single machine and into separate, scalable storage

Much of the commentary that followed read that claim through systems people already know. Four readings came up often:

“LTAP is CDC under another name”

“A single copy of the data can have only one physical layout, so either point lookups or scans must be slow”

“Iceberg or Delta tables on object storage are inherently stale”

“Postgres already provides the extraction path: logical replication”

Each of these myths rests on a wrong assumption about how Lakebase stores data. Let's take them one at a time.

Myth 1: “LTAP is replication”

The reality: LTAP keeps one durable copy of each table, in the lake. The layers in front of it hold some of the same bytes to make that table durable and fast, but none of them is a second database.

From the outside, LTAP can look like a familiar pipeline. Postgres writes rows, a process tails the log, the changes land in an Iceberg table, and analytics reads Iceberg. That pipeline is CDC. It is also the structure of logical replication, mirroring, and most zero-ETL products. So it is easy to classify LTAP as the same design.

To see why it isn't, it helps to be precise about what counts as a second copy. There are two different ways a system can end up holding the same data in more than one place: 

Replication keeps two databases aligned by shipping changes from one to the other. Each side has its own catalog and its own timeline, and either side can be queried as a source of truth. If shipping stalls, the two copies drift apart, and nothing forces them back together.

Tiering keeps one database and puts a stack of caches in front of it. The caches hold some of the same bytes so reads are fast. If a read misses the cache, it goes one layer down and costs some latency. It never creates a second catalog, and if a cache is lost, the table is not.

So the test is simple: can this layer be queried, or failed over to, as its own database? If yes, it is a replica. If no, it is a tier.

Replication keeps two databases in sync. LTAP keeps one database, with the lake as the durable tier and caches in front of it

By that test, Postgres has always been tiered. The WAL, the buffer cache, and the indexes all store bytes that overlap with the heap. Nobody calls an index a replica of the table, because it has no catalog or timeline of its own. 

LTAP keeps that same hierarchy. It moves it onto object storage and lets a lakehouse engine read the bottom, durable tier.

Here is what that hierarchy looks like in Lakebase, following a write and then a read:

A write is durable when its WAL reaches a quorum of safekeepers. The pageserver then applies the log, keeps hot pages in memory and on local disk, and materializes data into object storage.

A read starts at the compute node, which keeps a buffer cache and a local file cache. Indexes, TOAST, and visibility maps sit on the same hot path. A page that isn't cached there comes from the pageserver.

Object storage itself works the same way. When you PUT an object into S3, it is stored more than once, often across availability zones using erasure coding. The API still names one object, and no sync job runs between the copies. Lakebase stacks the same idea in front of the lake: log, caches, pageserver, object storage. The extra files are how one table is made durable and fast.

CDC does create a second database. It decodes a logical change stream into a foreign schema, usually keyed by primary key, and upserts into a table the source never reads back. That second table can be queried on its own, and it can fall out of step with the source: types can drift, DDL can go missing, a replication slot can drop rows, and tables nobody selected never appear. The result is two catalogs.

LTAP has no second table to keep in step. There is no list of tables to opt into. A table that exists in Postgres exists in the lake, and the pages the primary serves are caches of that store.

Myth 2: “A single copy means slow lookups or slow scans”

The reality: two lossless encodings of the same rows can serve different access paths - heap pages for point lookups, Parquet for scans. Each query reads the layout that's fast for it.

Even if the extra files are a hierarchy, layout remains a separate question. Point lookups are efficient on rows, scans are efficient on columns. If there is only one encoding on disk, one of those access patterns pays in I/O. Calling the table a single logical copy does not change that.

That reasoning is correct, but it rests on one assumption - that one copy implies one encoding. It doesn't, as long as two formats can represent the same rows without loss. 

So the question is whether LTAP's two formats are lossless:

The primary format is Postgres heap pages, which are well suited to point lookups and updates.

The second format is built for analytical access: Parquet behind Delta or Iceberg. Values are preserved, and the files carry enough physical metadata (heap block, offset, and MVCC versions) that the original pages can be rebuilt. Values that Parquet cannot represent losslessly are not coerced or dropped; they travel in an overflow field in the same table.

Because nothing is lost, the columnar copy can be the durable store in the lake, and the heap pages become a cache of it. Each access path then reads the format that suits it. Lookups hit 8 KB pages on the compute node, on local disk, and on the pageserver. Scans hit Parquet through an OLAP engine. Indexes remain in the hot tier; they are not transcoded into Parquet.

Postgres already works this way. It keeps two physical structures over the same rows for different access paths: a heap and a covering index. Nobody treats the index as a CDC replica of customer. SELECT * FROM t WHERE id = 1 and SELECT count(*) FROM t are two access paths over one table.

Point lookups read heap pages and scans read Parquet. The pages can be rebuilt from the Parquet, so both hold the same row

A narrower observation is valid. The columnar files from Postgres are heap-ordered: rows sit in page-number order, because the page is the unit rewritten on each compaction. That layout is efficient for reconstructing a page and for rewriting only dirty pages. It is a poor clustering key for WHERE order_date >= …. If a warehouse query needs that clustering, it can be derived the way silver tables are already derived from any operational source. What is no longer required is a CDC pipeline whose purpose was to create the lake copy.

Myth 3: “Open table formats mean eventual consistency”

The reality: every analytical query reads a consistent snapshot. Commits that haven't reached the lake yet are read from the pageservers, so nothing committed before the query is missed.

Iceberg and Delta provide snapshot isolation on object storage, and object storage has high latency. So it is natural to conclude that analytics is always behind, that many small tables cannot stay fresh, and that a join across tables cannot be consistent. That description fits CDC ingestion into a lake.

It depends on one assumption: that a query can only see what has already landed in object storage. In LTAP, a query can also see pages that are not in object storage yet. That changes the answer, and it is easiest to see by following a query.

When an analytical query starts,

It asks Postgres for the current LSN. That is a metadata lookup.

It reads nearly all of the data from object storage: everything already materialized up to that LSN. An engine that does not share CPU with the OLTP node does this read.

It merges the rest from the pageserver. Pages that got committed after the last materialization come from the pageserver and are merged on top.

Postgres serves none of the bulk read traffic - it returns a number and, at most, those recent pages. The lake does not need to contain the latest commit for that commit to be visible. It needs to contain almost everything, plus a way to fetch pages that got committed after the last materialization. The LSN is the cutoff: it tells the engine how far the merge has to go.

The same LSN also keeps joins consistent. CDC applies table by table, so one relation can sit at a later slot position than another, and a join mixes two points in time. LTAP reads every table at a single LSN, the same kind of cutoff Postgres uses for REPEATABLE READ, rather than at whatever each pipeline last flushed. That LSN is a shared watermark. APIs for reading recent changes since a watermark are being developed both natively for Lakebase and as an extension for upstream Postgres.

Parquet supplies the bulk of the table, and the pageservers fill in recent commits up to the LSN

Small tables are a metadata cost, not a freshness limitation. Iceberg's overhead is files and snapshots. For a relation of a handful of rows, that bookkeeping costs more than it saves, and layout does not matter for scan performance. Those tables stay in the page cache, participate in the LSN-consistent read, and skip the Iceberg commit.

This leaves two claims that sound alike but aren't:

"A long query doesn't see new commits" - true, and correct. A query that started at LSN X will not see commits after X. That is snapshot isolation, the same contract as REPEATABLE READ, not replica lag.

"A query misses committed data" - this would be an incorrect result: a query that started at X and missed a commit that was already durable at X, because it was still sitting in a CDC buffer. Merging recent pages as of that LSN is how LTAP avoids this case.

The materialized columnar snapshot can lag behind the primary by the most recent compaction interval, but the query itself does not. The fact that Parquet files may be several minutes behind does not mean a reader is unable to see committed data.

Myth 4: “Postgres replication is all we need”

The reality: logical replication is a costly way to keep the lake in sync with the database. It replays every change, row by row, and needs a pipeline that has to keep up. Decoding the WAL in parallel doesn't get around this, because log records can't be processed independently.

A standard way to get analytics on fresh Postgres is to enable logical replication. Debezium, PeerDB, pgoutput, and similar tools start from the observation that the database already emits a change stream, and ask what storage format is needed in addition.

The question is whether a change stream is a good way to build a table in the lake. Two properties of the stream say it isn't: how much it has to read, and how much state it has to carry.

Logical replication reads every change

The WAL is a physical log of page edits, written so Postgres can crash and recover. Logical decoding turns it into row events in commit order. That conversion cannot collapse many updates to the same page into the current image of the page; every change has to pass through. On a hot page, Postgres may log the same 8 KB a hundred times between checkpoints. Logical decoding emits a hundred events, then a downstream job upserts them into Parquet by primary key. In contrast, a page-tiering path reads the page once. In the write-intensive workloads we have measured, CDC reads at least the WAL volume, while dirty-page compaction may read a fraction of it.

Logical replication carries state

A replication slot is a cursor into WAL that the consumer must keep advancing. If the consumer falls behind, WAL remains on disk until the slot is dropped or the disk fills. Logical decoding does not emit DDL, so ALTER TABLE, type changes, and renames either make events inconsistent with the current catalog or invalidate the slot. Tables without a primary key need REPLICA IDENTITY FULL. Because the pipeline is expensive, it is typically applied to a subset of tables. That is the opposite of every Postgres table appearing in the lake because it exists.

A related claim: logical decoding can be skipped and the WAL decoded in parallel

Pageservers already consume WAL, so a parallel log-to-Parquet path can look like a small additional step. It is possible in principle. It has not been built as a production extractor, and it is not the path LTAP uses. WAL records are ordered because later ones assume earlier ones. Visibility lives in pg_xact, the proc array, and the catalog at that LSN, not in the tuple. Independent workers cannot each decode a disjoint byte range and emit correct current rows.

What LTAP parallelizes is pages, not log records. The system asks which pages changed since the last snapshot (WAL summaries on Postgres 17+, or the pageserver's log-structured merge trees), and each dirty page is independent work. That set is partitioned, the pages are transcoded, and one Iceberg snapshot is committed. The log is how writes were made durable, and one way to discover which pages changed. The snapshot is the current pages, not a replay of every record that dirtied them.

None of this makes replication the wrong tool for its intended jobs: migrations, major version upgrades, and cutovers to a second Postgres. But using logical replication as storage produces a second source of truth.

All four myths come from the same misunderstanding: treating every materialization of the data as a second database.

LTAP keeps one logical table, and the lake is its durable copy. Heap pages, indexes, and caches are how Postgres serves that table at OLTP latency, much as a covering index serves queries over a heap. An OLAP engine scans the Parquet directly and merges in transactionally consistent changes since the last materialization.

None of these layers adds a second catalog or a second source of truth that can drift.

For a deep dive on how Lakebase's storage architecture leads to LTAP, see this post.

Get the latest posts in your inbox

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 Databricks Blog(Dispelling LTAP myths),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗