Eclipse Aerospace 借 Databricks 简化飞行数据接入

降低成本与复杂性:消除了冗余存储和多步骤处理系统,减少运营开销并简化管理。
随着航空航天、制造业、生命科学以及其他数据密集型行业的企业推进运营现代化,它们正将更多传感器数据、遥测数据和科学数据引入数据平台。但这些数据大部分仍难以规模化使用,因为它们以专用或专有格式存在,主流分析引擎无法直接读取。这给行业带来一个现实挑战:团队不得不在几种做法之间做选择——构建缓慢且内存密集型的自定义转换、维护专用连接器,或者在数据可被分析之前增加额外的解码阶段。
PySpark 自定义数据源提供了一条更简单的路径。它让团队能够用 Python 将专用格式引入 Spark,而无需从零构建独立的连接器。对于超轻型喷气机制造商和运营商 Eclipse Aerospace 而言,与 Databricks 合作提供了一条途径,让近二十年的专有飞行数据能够更方便地被服务中心和工程师使用,同时无需增加昂贵的常开型基础设施。Databricks 带来了无服务器数据平台和 Spark 集成,而 Eclipse Aerospace 则贡献了其现有的飞行数据解码器和航空航天专业知识。
Eclipse Aerospace 与 Databricks 共同开发出一条从飞行数据到分析的更直接路径,降低了此前多阶段流程的延迟、重复和运营开销,同时保留了 Databricks Serverless 的灵活性。
Eclipse Aerospace 的数据接入挑战
Eclipse Aerospace 依托 Databricks 存储近 20 年的飞行数据,总量超过 3TB。它依靠来自超过 100 万飞行小时的数据为服务中心和工程师提供支持,这些数据从多个来源采集并经过验证,包括:近实时卫星遥测数据;完整飞行历史的大批量上传;以及外部数据源。其所有遥测数据都以专有文件格式存储,需要自定义解码器,而这些解码器已用多种语言开发完成,包括 Rust 和 Scala。
在 Databricks Runtime 15.4 LTS 提供 PySpark 自定义数据源之前,Eclipse Aerospace 团队有三种可行方案,可以将其专有文件类型解码并加载到 Databricks:
将文件以二进制类型加载,并使用 Python 用户定义函数(UDF)进行拆分
在专用计算资源上使用 Scala 编写的自定义解码器
在加载到 Databricks 之前,先将文件解码为一种中间文件类型
对于简单的自定义文件类型,UDF 是自然而然的选择。它们以性能换取可读性。然而,当消息格式长度可变时,UDF 存在真实的风险:在处理长消息时内存耗尽,因此无法使用。现有的 Scala 解码器可用于实现原生 Data Source v1/v2,但在 Databricks 中使用它需要专用计算资源。用户期望能够快速上传和查看数据,这就要求该计算资源始终保持开启,其成本高于其他无服务器方案。于是团队只剩下最后一个选择:通过 AWS Lambda 上基于 Rust 的解码器,将文件解码为一种中间文件格式,在这里即 Avro。
该方案成本较低,但它:
因多跳摄取模式而引入了延迟
Lambda 解码器与下游消费 Avro 输出的 Spark 作业之间重复了 schema 逻辑
这让团队失去了流水线的可观测性;Spark UI 和作业指标所能看到的范围只从 Avro 落地区开始,因此工程师在排查缓慢或失败的摄取任务时,不得不去平台另一个完全不同的地方查找 Lambda 日志
意味着要同时为存储原始二进制文件及其 Avro 副本付费
为什么 Databricks 与 PySpark 自定义数据源至关重要
Databricks 已经是 Eclipse Aerospace 用于存储和处理飞行数据的平台。其决定性优势在于 Spark 原生执行、PySpark 自定义数据源与 Databricks Serverless 三者的结合。Databricks Serverless 会自动分配并管理按需算力,因此 Eclipse Aerospace 无需自行配置或持续运行专用基础设施。PySpark 自定义数据源使得无需构建 JVM 连接器,就能封装已有的 Rust 解码器。由于解码现在在 Spark 内部运行,同一平台即可提供 Spark UI、作业指标以及下游 Delta 工作流,而不必将关键的摄取环节交给单独的 Lambda 流水线。
大多数数据工程依赖常见的数据格式,但一些最有价值的数据仍隐藏在 PySpark 无法直接访问的格式中,例如用于科学数据的 HDF5,以及航空航天数据所用的 ARINC 429 等行业标准。Databricks Serverless 为团队提供了一种灵活方式,可大规模释放这些数据的价值,而基于 Rust 的实现则能带来超越纯 Python 方案的性能。
PySpark 自定义数据源让你可以用纯 Python 定义一种可读写的文件格式,并像使用任何其他数据源一样使用它。它的主要优势是简单:用户可以用纯 Python 定义读写函数,还能定义并行化等重要特性。Eclipse Aerospace 团队更进一步:通过构建一个由 Rust 支持的 Python 包,其基准测试性能是同类基于 PyArrow 的纯 Python 数据源的两倍。与不使用 Arrow 的原生 Python 实现相比,其性能提升了近 20 倍。
下面的示例代码仓库逐步演示了如何为 ARINC 429 实现一个自定义数据源。随附的代码包含超出本文范围的设计决策,以及你可以自行运行的基准测试。
如何借助 PySpark 自定义数据源让这一问题变得可解
Apache Spark 4.0 发布了 Python Data Source API(在 Databricks 文档中被称为 PySpark 自定义数据源),它让你可以为批处理或流式数据注册自定义格式,例如:
该 API 在 Apache Spark 4.0 中引入并可在 Databricks 上使用,自 Databricks Runtime 15.4 LTS 起正式可用,同时也支持 Serverless environment 2 及更高版本。由于该 API 用 Python 实现,因此不需要单独的 JVM 连接器或 JAR。
该 API 是为创建原生数据源而提供的简化接口,用户只需实现几个类,并通过以下方式注册它们:
这些解码器有两种形式:原生 Python 和 Apache Arrow 格式,后者性能要高得多。Apache Arrow 是一种「面向扁平与嵌套数据的、与语言无关的列式内存格式」,因此非常适合充当不同语言之间共享数据对象的接口。该 API 正是利用这一点,允许用户直接返回 pyarrow.RecordBatch 对象,Spark 会将它们按批消费,而不必逐行物化成一个 InternalRow。
这引出了一个问题:能否让 Python 充当控制层,由 Rust 承担解码的重活,同时全程保持 Apache Arrow 格式?
解决方案架构概览
这背后的架构是单一逻辑连接器,其解码路径几乎全部由编译后的代码构成,而 Python 层足够薄,对 Eclipse Aerospace 现有的 Rust SDK 改动极小。这个薄 Python 层使用 PyO3/Maturin 构建为独立包,导入到管道中,像其他任何数据源一样使用。为确保最佳性能与关注点分离,团队遵循了几条架构原则。
解码器并不知道 Spark 的存在。核心 Rust crate 不依赖 PyO3、Arrow 或 Spark,只接收字节,返回结构化记录。这让 Eclipse Aerospace 得以基本原封不动地移植现有解码逻辑,因为 SDK 无需为此重写,只需包装即可。
Arrow 是唯一跨越外部函数接口(FFI)边界的东西,而且它在跨越时不进行复制。Rust 胶合层使用 arrow-rs 构建一个 Arrow RecordBatch,然后通过 Arrow C Data Interface 将其作为 pyarrow.RecordBatch 交给 Python。在此过程中没有序列化步骤,因为不会重新编码任何字节,也不会构建任何逐行的 Python 对象。Rust 分配的缓冲区就是 Python 读取的缓冲区。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Eclipse Aerospace simplifies flight-data ingestion with Databricks)
Reduced costs & complexity: Eliminated redundant storage and multi-step processing systems, cutting operational overhead and simplifying management.
As organizations in aerospace, manufacturing, life sciences, and other data-intensive industries modernize their operations, they are bringing more sensor, telemetry, and scientific data into data platforms. But much of this data remains difficult to use at scale because it lives in specialized or proprietary formats that mainstream analytics engines cannot read directly. This creates a practical industry challenge: teams must choose between building slow, memory-intensive custom transformations, maintaining specialized connectors, or adding extra decoding stages before the data can be analyzed.
PySpark custom data sources offer a simpler way forward. They let teams bring specialized formats into Spark using Python, without building a separate connector from scratch. For Eclipse Aerospace, a very light jet manufacturer and operator, partnering with Databricks offered a way to make nearly two decades of proprietary flight data more accessible to service centers and engineers without adding costly, always-on infrastructure. Databricks brought the serverless data platform and Spark integration, while Eclipse Aerospace contributed its existing flight-data decoders and aerospace expertise.
Together, Eclipse Aerospace and Databricks developed a more direct path from flight data to analytics, reducing the latency, duplication, and operational overhead of the previous multi-stage process while preserving the flexibility of Databricks Serverless.
Eclipse Aerospace's data ingestion challenge
Eclipse Aerospace relies on Databricks to store nearly 20 years of flight data, totalling more than 3TB. It supports its service centers and engineers with data from more than a million flight hours, sourced and verified across multiple sources, including: near-real-time satellite telemetry; large bulk uploads of full flight histories; and external data sources. All of its telemetry is stored in proprietary file formats that require custom decoders, which are already developed in multiple languages, including Rust and Scala.
Before the Pyspark custom data sources was available in Databricks Runtime 15.4 LTS, the Eclipse Aerospace team had three viable options for decoding and loading its proprietary file type into Databricks:
Load the files as binary types and split using Python user-defined functions (UDF)
Use a custom decoder in Scala on dedicated compute
Decode files into an intermediary file type prior to loading in Databricks
For simple custom file types, UDFs are a natural option. They trade performance for readability. However, where the message format has variable length, UDFs pose a real risk of running out of memory on long messages and could not be used. The existing Scala decoder could be used to implement a native Data Source v1/v2, but using it in Databricks required dedicated compute. Users expected to be able to upload and view data quickly, which would require keeping this compute to be always on, representing a larger cost than the other serverless options. This left the team with the final option of decoding the files into an intermediary file format, in this case, Avro, via a Rust-based decoder on AWS Lambda.
This solution was low-cost, but it:
Introduced latency from the multi-hop ingestion pattern
Duplicated schema logic between the Lambda decoder and the downstream Spark jobs that consumed the Avro output
Cost the team pipeline observability; the Spark UI and job metrics only ever saw the world starting at the Avro landing zone, so engineers debugging a slow or failed ingest had to go find Lambda logs in a completely different part of the platform
Meant paying to store both the original binary files and their Avro copies
Why Databricks and PySpark Custom Data Sources Matter
Databricks was already Eclipse Aerospace’s platform for storing and processing flight data. The deciding advantage was the combination of Spark-native execution, the Pyspark custom data source, and Databricks Serverless. Databricks Serverless automatically allocates and manages on-demand compute, so Eclipse Aerospace did not need to provision or keep dedicated infrastructure running. The PySpark custom data source made it possible to wrap the existing Rust decoder without building a JVM connector. Because decoding now runs inside Spark, the same platform provides the Spark UI, job metrics, and downstream Delta workflows instead of handing the critical ingest step to a separate Lambda pipeline.
Most data engineering relies on common data formats, but some of the most valuable data remains hidden in formats that PySpark cannot access directly, such as HDF5 for scientific data and industry standards such as ARINC 429 for aerospace data. Databricks Serverless gives teams a flexible way to unlock that data at scale, while Rust-backed implementations can deliver performance beyond a pure-Python approach.
PySpark custom data sources lets you define a read/writable file format in pure Python, usable like any other data source. Its main benefit is simplicity, letting users define functions to read and write in pure Python, as well as define important characteristics like parallelization. The Eclipse Aerospace team took this a step further: by building a Rust-backed Python package, it delivered twice the benchmarked performance of a similar PyArrow-based pure-Python data source. Compared with a native Python implementation without Arrow, it delivered nearly 20x the performance.
The following example code repository walks through a custom data source for ARINC 429. The accompanying code includes design decisions beyond the scope of this article, along with benchmark tests you can run yourself.
How this became solvable with the PySpark Custom Data Sources
Apache Spark 4.0 shipped a Python Data Source API (referenced in Databricks documentation as PySpark custom data sources) that lets you register a custom format for batch or streaming data like:
Introduced in Apache Spark 4.0 and available on Databricks, the API is generally available from Databricks Runtime 15.4 LTS and is also supported on Serverless environment 2 and later. Because the API is implemented in Python, it does not require a separate JVM connector or JAR.
The API is a simplified interface for creating native data sources, with the user implementing a few classes and registering them with:
These decoders come in two flavors: native Python, and Apache Arrow format, the latter being the much more performant choice. Apache Arrow is a "language-independent columnar memory format for flat and nested data," making it an ideal interface between different languages sharing data objects. The API takes advantage of this by allowing users to return pyarrow.RecordBatch objects directly, and Spark will consume them as batches rather than materializing one InternalRow per row.
This raised the question: could Python serve as the control layer, with Rust doing the heavy lifting of decoding while maintaining the Apache Arrow format throughout?
Solution architecture overview
The architecture behind this is a single logical connector with a decode path that's almost entirely compiled code, and a Python layer thin enough that it makes minimal change to Eclipse Aerospace's existing Rust SDK. This thin Python layer is built with PyO3/Maturin as a standalone package, imported into pipelines and used just like any other source. To ensure the best performance and separation of concerns, the team followed a few architectural principles.

The decoder doesn't know Spark exists. The core Rust crate has no dependency on PyO3, Arrow, or Spark and only takes bytes, returning structured records. That allowed Eclipse Aerospace to port existing decode logic largely unchanged, as the SDK did not need to be rewritten for this, only wrapped.
Arrow is the only thing that crosses the foreign function interface (FFI) boundary, and it crosses it without copying. The Rust glue layer builds an Arrow RecordBatch using arrow-rs, then hands it to Python as a pyarrow.RecordBatch via the Arrow C Data Interface. In doing this, there is no serialization step, as no bytes are re-encoded and no per-row Python objects are built. The buffers Rust allocated are the buffers Python reads.
The PySpark layer is intentionally lightweight. It parses options, decides how to partition (one InputPartition per input in the reference example), and defines the schema. In PySpark custom data sources, DataSource, DataSourceReader, and their streaming equivalents all have to be serializable, so Spark can pickle the reader instance and ship it to a worker. By making the PySpark layer lightweight, the DataSourceReader instance that Spark pickles and ships to a worker only ever holds plain Python state (file paths, an int for batch size). The PyO3-backed Rust object is constructed inside read(), after the reader has already arrived on the worker. It's never itself part of what gets pickled.
The native code ships as a Python wheel, built with Maturin. On Databricks that wheel is a build artifact, attached to a job or notebook environment as a normal Python dependency. Because the implementation is compiled, version and architecture compatibility require care, though a proper CI/CD process can manage this as demonstrated in the example repo.
How the connector runs in Databricks
On Databricks Serverless, the runtime path is straightforward:
A proprietary telemetry file lands in cloud storage.
A Databricks Spark read invokes the registered PySpark custom data source for the custom format.
Spark running on Databricks creates a serializable reader containing the input path and read options, then schedules input partitions on workers.
Each worker constructs the Rust decoder inside read() and passes the file bytes to the compiled decoder.
Rust returns Arrow RecordBatches through the Arrow C Data Interface, avoiding serialization and per-row Python objects.
Spark running on Databricks consumes the batches and writes the decoded data to downstream tables such as Delta.
Decode errors, bad records, Spark UI stages, and job metrics remain visible within the Databricks workflow.
This is the key difference from the previous design: the original file is decoded as part of the Spark read rather than being copied to an intermediary Avro landing zone first.
What impact did it have for Eclipse Aerospace
By replacing the multi-stage Lambda and Avro workflow with a single Spark-native read, Eclipse Aerospace made flight data available faster while reducing infrastructure, storage, and operational overhead. The new Arrow-native PySpark custom data sources connector also brings decoding into Spark, improving visibility into pipeline performance and allowing schema and data quality logic to be managed in one place.
Implementing the custom data source has significantly simplified Eclipse Aerospace’s infrastructure, and the effects show up in four distinct places: latency, cost, observability, and engineering time.
Latency. Collapsing a multi-hop pipeline (upload, Lambda, decode, write Avro, read Avro, load to Delta) into a single Spark-native read removes several network hops and several points where a file sits in a queue waiting for the next stage to pick it up. The new connector can process records within a minute of file upload including startup time and can commonly process large, multi-gigabyte batches in parallel in under two minutes due to parallelism.
Cost. Removing the Lambda hop and the Avro intermediary removed a whole tier of infrastructure Eclipse Aerospace had to run, monitor, and pay for. They no longer pay to store two copies of every flight, the original binary and its Avro conversion, which on almost two decades of retained flight history is a meaningful amount of duplicated storage. This means the majority of files remain in Glacier storage on AWS, with easily repeatable decodes if the original data is ever needed again.
Observability. Because the decode step now happens inside Spark itself, it shows up in the Spark UI and standard job metrics like any other stage of a job. Eclipse Aerospace uses custom "badRecordsPath," which mimic the CSV/JSON loader, to capture misformatted data, and maintains extensive data quality pipelines that capture statistics directly from the decoder.
Engineering time. Schema logic used to be defined twice: once in the Lambda decoder that produced Avro, and again in the downstream Spark jobs that consumed it. Any change to the message format meant updating both, in two different codebases, with the risk of the two definitions drifting apart. With a single, versioned connector, the schema is defined once, next to the decode logic itself. This also means the same architectural pattern, a thin Python DataSource wrapping a Rust decoder, is now reusable for the other proprietary formats Eclipse Aerospace supports, rather than something that has to be reinvented per format.
Eclipse Aerospace's experience shows how specialized data formats can be brought into Databricks without creating a separate ingestion platform around them. By keeping the decoder in Rust, using Python as the integration layer, and passing records through Apache Arrow, the custom data source reads original files directly in Spark and remains compatible with the flexibility of Databricks Serverless.
For teams working with proprietary or industry-specific formats, the reusable pattern is simple. Engineers can keep domain-specific decoding logic close to the source format, expose it through a PySpark custom data source, and let Spark handle partitioning, execution, observability, and downstream processing.
Looking to get started? Take a look at the following:
Frequently asked questions
What is a PySpark Custom Data Source?
A PySpark custom data source is a user-defined Spark data source implemented in Python. It lets teams read specialized file formats as part of a standard Spark workflow without building a separate JVM connector.
What data ingestion problem does a custom Arrow-native PySpark Custom Data Source connector solve?
The Arrow-native PySpark custom data source connector ingests proprietary flight telemetry into Databricks without relying on a multi-stage pipeline, dedicated always-on compute, or duplicate Avro copies of the original files.
What is Databricks Serverless?
Databricks Serverless is a fully managed compute service that lets you run workloads without provisioning or managing any infrastructure in your cloud account. Databricks automatically allocates, scales, and optimizes the compute resources for you.
How does Databricks Serverless connect to the Arrow-native PySpark Custom Data Source connector?
The connector uses the Pyspark custom data sources to run file-reading logic inside Spark on serverless compute. Python handles the control flow, while Rust and Apache Arrow handle high-performance decoding and data transfer. This allows proprietary files to be read directly without dedicated always-on compute or an intermediary Avro pipeline. PySpark custom data sources are available in Databricks Runtime 15.4 LTS and above, or serverless environment version 2.
Get the latest posts in your inbox
原文为英文,正文由自动抓取并保留原文。
原文发布于 Databricks Blog(Eclipse Aerospace simplifies flight-data ingestion with Databricks),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。