Apache Spark 与 Flink:该选哪个

对于超大规模键控状态(keyed state)或现有 Flink 平台上的复杂事件处理等特殊场景,Flink 仍然是合适的选择。
对大多数团队而言,Apache Spark™ 是更好的选择。Spark 在同一个引擎上运行批处理 ETL、SQL 分析、机器学习和流处理,而 Real-Time Mode 如今让 Spark 流处理达到毫秒级端到端延迟。Apache Flink 是流原生的引擎,适合专门的事件处理负载,但选择它通常意味着要在 Spark 之外再运行一个引擎。
过去,延迟差距是团队引入 Flink 的主要原因。有了 Real-Time Mode,这一理由已基本不复存在,真正的问题变成了你想构建、运维和治理多少个引擎。
Apache Spark 和 Apache Flink 有什么区别?
Spark 是一个统一的引擎,覆盖批处理、流处理、SQL 和 AI;Flink 则是一个专门的流处理器。两者都是 Apache 软件基金会的开源项目,都具备精确一次(exactly-once)处理、有状态算子和 SQL API。区别在于覆盖范围。
Spark 最初是批处理引擎,后来加入了 Structured Streaming,它把流视为一张不断增长的表。同一份 DataFrame 和 SQL 代码,既可以作为定时批处理作业运行,也可以作为微批流或 Real-Time Mode 流运行。Flink 则从一开始就为流处理而设计,把批处理视为有界流。
Spark 和 Flink 在具体功能上如何逐项对比?
下表按照通常决定选型的几个维度对这两个引擎进行比较。
统一:批处理、流处理、SQL、机器学习与 AI 数据准备
流优先,将批处理视为有界流
微批处理下为秒级,实时模式下为毫秒级
批处理与大规模 ETL
PB 级批处理的行业默认选择
受支持,但实践中较少见
Python、SQL、Scala、Java,以及通过 Spark Connect 支持 Go、Rust 和 Swift 客户端
Java、SQL、Python(PyFlink)
聚合、去重、连接、transformWithState、RocksDB 状态存储
Flink 2.x 中的键控状态、定时器、分离式状态(ForSt)
复杂事件处理
使用 transformWithState 实现自定义逻辑
内置 CEP 库和 SQL MATCH_RECOGNIZE
MLlib、特征管道、SQL 中的向量搜索、LLM 数据准备
原生 ML 能力有限,通常需与另一系统配合使用
原生 Delta Lake 和 Apache Iceberg 读写
Iceberg 及其他格式的连接器
Databricks、AWS EMR 和 Glue、Google Cloud、Microsoft Fabric
AWS、Confluent、Ververica 等
表 1. Apache Spark 与 Apache Flink 对比。来源:Apache Spark Real-Time Mode 文档、Apache Flink 2.0 发布说明和 Flink CEP 文档。
在流处理方面,Spark 比 Flink 更快吗?
在 Databricks 公布的基准测试中,Spark Real-Time Mode 在无状态任务上与 Flink 持平,并在有状态聚合和流-静态表连接上具有更低的 p99 延迟。该基准测试代码公开,使用源自客户的特征计算工作负载,以每秒 10,000 条记录的速率运行 30 分钟,测量 Kafka 输入到输出的延迟。Databricks 将结果总结为比 Flink 最高快 92%。
图 2. 每秒 10,000 条记录下的 p99 端到端延迟。Spark Real-Time Mode 显示其最差运行结果,Flink 显示其最佳运行结果。
Spark Real-Time Mode p99
无状态转换
如何解读这些数字:该基准测试由 Databricks 运行。Spark 运行在 Databricks 上,Flink 运行在 AWS Managed Service for Apache Flink 1.18 上。结果比较的是 Spark 最差的一次运行与 Flink 最好的一次运行。请把它们视为针对这些负载模式的证据,在做决定前先测试你自己的流水线。
生产环境的结果也印证了同样的结论。MakeMyTrip 曾考虑 Flink,最终选择 Spark Real-Time Mode,以避免引入第二个引擎,并将 P50 延迟降至 44 ms,P99 从一分多钟降至约 500 ms。Coinbase 和 DraftKings 也在生产环境中运行 Real-Time Mode。关于该引擎如何实现这一点,请阅读《Breaking the microbatch barrier》。
如何在 Spark 中获得接近 Flink 的延迟?
答案是改变触发器。Real-Time Mode 是作用于现有 Structured Streaming 查询的一个触发器,因此同一套代码无需重写即可在不同延迟层级之间迁移。
Real-Time Mode 需要 update 输出模式,并从 Kafka 或行级 foreach sink 读取和写入数据。有状态算子将在 Apache Spark 4.3 中进入开源版本,目前在 Databricks 上已正式可用。在 Databricks 上,它可运行于 classic compute 或 Lakeflow Spark Declarative Pipelines 中。完整支持矩阵请参阅 Real-Time Mode 参考文档。
面向企业的智能体式 AI 实战指南
什么时候应该选择 Apache Flink 而不是 Spark?
对于少数专门的流处理工作负载,Flink 可能是更合适的选择。当以下多数条件成立时,可以考虑它:
超大规模的键控状态。Flink 2.x 的分离式状态(disaggregated state)正是为远超本地磁盘容量的状态而构建的。
复杂事件处理。你需要用 Flink CEP 或 SQL MATCH_RECOGNIZE 在事件序列中进行模式匹配。
低延迟路径上的事件时间定时器。你的逻辑高度依赖事件时间定时器,而 Spark Real-Time Mode 在 transformWithState 中尚未支持这一特性。
已有的 Flink 平台。你已经在生产环境运行 Flink,并且该数据流不供给批处理、SQL 或机器学习。
如果该数据流同时还要供给分析、机器学习特征或湖仓表,那么 Spark 可以避免在两套引擎之间重复逻辑、运维和治理。
为什么只运行一套引擎很重要?
每多一套引擎,就多一套代码库、多一套技能组合和多一套治理规则。将 Spark 用于批处理、Flink 用于流处理的团队,往往要把同一套业务逻辑重写两遍,还要排查两者之间的偏差。
一套代码库。相同的 DataFrame 和 SQL 逻辑既可以按批处理、微批处理运行,也可以按 Real-Time Mode 运行。
一个团队。Python 和 SQL 开发者无需学习新的运行时,就能构建实时管道。
一套治理模型。流式表和批处理表在 Unity Catalog 中共享数据血缘与访问控制。
一条通往 AI 的路径。同一批管道可为训练和在线推理构建机器学习特征,正如 DraftKings 借助 transformWithState 所做的那样。
管道也可以采用声明式定义,而无需手工连线。Spark Declarative Pipelines 负责管理依赖关系以及流式表的增量刷新。
在 Databricks 上运行 Spark 流处理
Databricks 由 Apache Spark 的原始创建者创立,其 Real-Time Mode 已作为正式可用功能提供,其中包括有状态算子和 transformWithState。它新增了 Lakeflow Spark Declarative Pipelines、Unity Catalog 治理以及无服务器管道。为开源 Spark 编写的代码无需改动即可在 Databricks 上运行。如需入门,请参照 Apache Spark 教程:Real-Time Mode。
Apache Spark 比 Apache Flink 更快吗?
在 Databricks 公布的基准测试中,Spark Real-Time Mode 在无状态工作负载上与 Flink 持平,并在有状态聚合和流-静态表连接上具有更低的 p99 延迟。结果取决于具体工作负载,因此请对自己的管道进行基准测试。
Spark 流处理仍然是微批吗?
不只是。Spark Structured Streaming 默认采用微批模式,而 Real-Time Mode 会以毫秒级延迟持续处理记录。你可以通过设置 trigger 为每个查询单独选择。
如果我已经在使用 Spark,还需要 Flink 吗?
对大多数工作负载来说不需要。Real-Time Mode 让 Spark 在你用于批处理、SQL 和 ML 的同一个引擎上实现毫秒级延迟,因此你只需运行一套系统,而不是两套。
什么时候 Flink 是更好的选择?
对于超大键控状态、复杂事件处理等专门工作负载,或者已经为独立流处理运行 Flink 平台的团队,Flink 可能更合适。
我能把 Flink 作业迁移到 Spark 吗?
是的。大多数 Flink SQL 和 DataStream 逻辑都可以对应到 Structured Streaming 中的 Spark SQL 和 DataFrames,而 Real-Time Mode 则覆盖低延迟路径。先从无状态作业和聚合作业入手,再把有状态逻辑迁移到 transformWithState。
在收件箱中获取最新文章
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Apache Spark vs Apache Flink: which should you use?)
Flink remains a fit for specialized cases such as very large keyed state or complex event processing on an existing Flink platform.
For most teams, Apache Spark™ is the better choice. Spark runs batch ETL, SQL analytics, machine learning and streaming on one engine, and Real-Time Mode now brings Spark streaming to millisecond, end-to-end latency. Apache Flink is a stream-native engine that fits specialized event-processing workloads, but choosing it usually means running a second engine next to Spark.
The latency gap used to be the main reason teams added Flink. With Real-Time Mode, that reason has largely gone away, and the real question becomes how many engines you want to build, operate and govern.
What is the difference between Apache Spark and Apache Flink?
Spark is a unified engine for batch, streaming, SQL and AI, and Flink is a specialized stream processor. Both are open source Apache Software Foundation projects with exactly-once processing, stateful operators and SQL APIs. The difference is scope.
Spark started as a batch engine and added Structured Streaming, which treats a stream as a table that keeps growing. The same DataFrame and SQL code runs as a scheduled batch job, a micro-batch stream or a Real-Time Mode stream. Flink was designed for streaming first and treats batch as a bounded stream.
How do Spark and Flink compare feature by feature?
This table compares the two engines on the dimensions that usually decide the choice.
Unified: batch, streaming, SQL, ML and AI data prep
Stream-first, with batch as a bounded stream
Seconds with micro-batch, milliseconds with Real-Time Mode
Batch and large-scale ETL
Industry default for petabyte-scale batch
Supported, less common in practice
Python, SQL, Scala, Java, plus Go, Rust and Swift clients via Spark Connect
Java, SQL, Python (PyFlink)
Aggregation, deduplication, joins, transformWithState, RocksDB state store
Keyed state, timers, disaggregated state (ForSt) in Flink 2.x
Complex event processing
Custom logic with transformWithState
Built-in CEP library and SQL MATCH_RECOGNIZE
MLlib, feature pipelines, vector search in SQL, LLM data prep
Limited native ML, usually paired with another system
Native Delta Lake and Apache Iceberg read and write
Connectors for Iceberg and other formats
Databricks, AWS EMR and Glue, Google Cloud, Microsoft Fabric
AWS, Confluent, Ververica and others
Table 1. Apache Spark and Apache Flink compared. Sources: Apache Spark Real-Time Mode docs, Apache Flink 2.0 release and Flink CEP docs.
Is Spark faster than Flink for streaming?
In Databricks' published benchmark, Spark Real-Time Mode matched Flink on stateless work and had lower p99 latency on stateful aggregation and stream-static joins. The benchmark code is public, uses customer-derived feature-computation workloads and runs at 10,000 records per second for 30 minutes, measuring Kafka input-to-output latency. Databricks summarized the results as up to 92% faster than Flink.
Figure 2. p99 end-to-end latency at 10,000 records per second. Spark Real-Time Mode shows its worst run and Flink shows its best run.
Spark Real-Time Mode p99
Stateless transformation
How to read these numbers: Databricks ran the benchmark. Spark ran on Databricks and Flink ran on AWS Managed Service for Apache Flink 1.18. The results compare Spark's worst run with Flink's best run. Treat them as evidence for these workload patterns, and test your own pipeline before deciding.
Production results tell the same story. MakeMyTrip considered Flink, chose Spark Real-Time Mode to avoid a second engine and cut P50 latency to 44 ms and P99 from more than a minute to about 500 ms. Coinbase and DraftKings also run Real-Time Mode in production. For how the engine achieves this, read Breaking the microbatch barrier.
How do I get Flink-like latency in Spark?
You change the trigger. Real-Time Mode is a trigger on an existing Structured Streaming query, so the same code moves between latency tiers without a rewrite.
Real-Time Mode needs update output mode and reads from and writes to Kafka or a row-level foreach sink. Stateful operators arrive in open source with Apache Spark 4.3 and are generally available on Databricks today. On Databricks, it runs on classic compute or in Lakeflow Spark Declarative Pipelines. See the Real-Time Mode reference for the full support matrix.
The agentic AI playbook for the enterprise
When should I choose Apache Flink over Spark?
Flink can be the better fit for a small set of specialized streaming workloads. Consider it when most of these are true:
Very large keyed state. Flink 2.x disaggregated state is built for state far larger than local disk.
Complex event processing. You need pattern matching across event sequences with Flink CEP or SQL MATCH_RECOGNIZE.
Event-time timers on the low-latency path. Your logic depends heavily on event-time timers, which Spark Real-Time Mode does not yet support in transformWithState.
An existing Flink platform. You already run Flink in production and the stream doesn't feed batch, SQL or ML.
If the stream also feeds analytics, ML features or lakehouse tables, Spark avoids duplicating logic, operations and governance across two engines.
Why does running one engine matter?
Every second engine adds a second codebase, a second skill set and a second set of governance rules. Teams that pair Spark for batch with Flink for streaming often rewrite the same business logic twice and debug drift between them.
One codebase. The same DataFrame and SQL logic runs as batch, micro-batch or Real-Time Mode.
One team. Python and SQL developers build real-time pipelines without learning a new runtime.
One governance model. Streaming tables and batch tables share lineage and access control in Unity Catalog.
One path to AI. The same pipelines build ML features for training and online inference, as DraftKings does with transformWithState.
Pipelines can also be declared instead of hand-wired. Spark Declarative Pipelines manage dependencies and incremental refresh for streaming tables.
Running Spark streaming on Databricks
Databricks was founded by the original creators of Apache Spark and runs Real-Time Mode as a generally available feature, including stateful operators and transformWithState. It adds Lakeflow Spark Declarative Pipelines, Unity Catalog governance and serverless pipelines. Code written for open source Spark runs on Databricks unchanged. To get started, follow the Apache Spark tutorial: Real-Time Mode.
Is Apache Spark faster than Apache Flink?
In Databricks' published benchmark, Spark Real-Time Mode matched Flink on stateless workloads and had lower p99 latency on stateful aggregation and stream-static joins. Results depend on the workload, so benchmark your own pipeline.
Is Spark streaming still micro-batch?
Not only. Spark Structured Streaming defaults to micro-batch, and Real-Time Mode processes records continuously with millisecond latency. You choose per query by setting the trigger.
Do I need Flink if I already use Spark?
For most workloads, no. Real-Time Mode gives Spark millisecond latency on the same engine you use for batch, SQL and ML, so you run one system instead of two.
When is Flink the better choice?
Flink can fit specialized workloads such as very large keyed state, complex event processing or teams that already run a Flink platform for standalone streams.
Can I migrate a Flink job to Spark?
Yes. Most Flink SQL and DataStream logic maps to Spark SQL and DataFrames in Structured Streaming, and Real-Time Mode covers the low-latency path. Start with stateless and aggregation jobs, then move stateful logic to transformWithState.
Get the latest posts in your inbox
原文为英文,正文由自动抓取并保留原文。
原文发布于 Databricks Blog(Apache Spark vs Apache Flink: which should you use?),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。