11-19 更新 实时
首页/ 技术栈/ 正文

Exa 借 AWS AI 芯片提升嵌入吞吐 50%

技术栈 来源:AWS Machine Learning 2026-11-19 07:58 0 阅读 机器翻译
Exa 借 AWS AI 芯片提升嵌入吞吐 50%

在 Exa,我们通过聚合并提供来自全网以及私有和合成来源的知识,构建着少数真正的网络索引之一。我们针对语义搜索优化该索引,让智能体和人类都建立在真实、高质量的数据之上。我们使用多塔语义机器学习(ML)模型,把网络数据和查询数据嵌入到同一向量空间,因此用户既能用自然语言表达复杂查询,也能使用传统关键词搜索。支撑这一切的关键技术是自研的向量数据库。它必须尽可能多地摄取高质量网络数据,并将它们全部嵌入。因此,嵌入的单位经济性是我们一切工作的基础。在数百亿乃至数千亿文档的规模下,找到每美元性能最佳的硬件并不是一个调优细节。任何单文档成本的改善都会累积成一个可观的数字。

本文详细介绍了我们如何在 AWS Neuron 硬件(AWS Inferentia2 和 AWS Trainium2)上优化嵌入基础设施,以实现更便宜、更灵活的推理工作负载,以及智能体式开发工作流如何推动了这一优化。

推理运行在异步流水线上,但必须做到高吞吐且快速,才能迅速服务最新数据,并缩短从发现到可服务之间的时间。

图 1:Exa 技术栈的简化视图

每一份被我们评定为质量足够高的结构化网络文档,都必须先被嵌入,然后才能对外提供。此图展示了原始网络数据如何从爬虫进入由解析、审核与过滤,再到嵌入组成的异步流水线,结果最终从向量数据库提供。

我们的 Neuron 硬件系列经验始于一次回填。转向更优的文档模型意味着要重新嵌入我们现有的语料库。我们选择了 300 亿份高优先级文档进行重新嵌入。这样的迁移会暂时使嵌入负载翻倍,这恰好成为重新评估硬件选择的绝佳时机。

图 2:回填带来的翻倍推理负载

上图展示了两个并行流水线,它们共享抓取和解析阶段:一个使用线上模型嵌入到线上索引,另一个使用新模型将优先文档重新嵌入到新索引,这正是负载翻倍的原因。这种翻倍明确促使我们重新考虑硬件和优化选择。

AWS Inferentia2 的数据指的是在最小实例类型 Inferentia2 xlarge 实例(inf2.xlarge)上的性能,该实例配备一个芯片和两个 NeuronCore。我们的工作负载运行在单个 NeuronCore 上,因此我们运行 Kubernetes Pod,将多个 Pod 打包到更大的实例上,而不让任何核心闲置,从而在整个嵌入机群中实现容器同质性。同样的流程运行在 Trainium2 NeuronCore 上,并通过数据并行以相同方式扩展。我们最初的 Inferentia2 实现是对模型进行直接的数据并行 FP16 表示。从这一基线开始,我们借助 Neuron SDK 和 AWS Trainium 团队的支持,一路将吞吐(docs/sec)尽可能推高。下图展示了每个里程碑上的吞吐提升。

图 3:各优化里程碑上的吞吐提升

从最朴素的编译出发,一系列逐步推进的优化最终带来了约 1.5 倍的吞吐提升。第一步是权重布局优化,它让权重布局与整体模型的调度和布局保持一致,从而消除了不必要的转置。我们还启用了 autocast,让编译器把矩阵乘法降级为更低精度,同时保留动态范围。降精度后的输出通过了我们 0.999 余弦相似度的正确性门槛(与参考实现相比,见 Agentic 方法论),因此精度影响落在我们可接受的阈值之内。

Neuron Kernel Interface(NKI)库为常见模型层和运算提供了经过优化的内核。这些预构建内核在常见特性上能带来立竿见影的收益,我们借助它们快速消除了最显而易见的一些低效环节。一个可直接替换的注意力内核为我们带来了下一轮性能提升。

下一个瓶颈是内存溢出。Neuron Explorer 的剖析结果持续显示,存在大量从高带宽内存(HBM)溢出并重新加载的情况,这使得 Tensor Engine 在前向传播过程中无法被充分利用。我们以一个可直接替换的 MLP 内核为跳板,剖析预构建实现的不足之处,随后编写了自定义的权重驻留融合 SwiGLU MLP 内核,消除了 HBM 停顿。

我们在布局操作、循环和内存操作上使用更高级的 NKI 语言,只有在特定张量运算时才下沉到更低层的 NKI ISA。把内核的大部分保留在高级语言中,能让工程师更容易理解,也更容易调试。

到这里我们撞上了一堵墙。在另一个片上核心空闲时测得的单加速器吞吐,明显高于两个核心同时运行时的水平。这一差距意味着,我们的理论数据并行吞吐本应远高于实际实现的总性能,而这指向了核心之间的资源争抢。

性能剖析与实验表明,我们的工作负载尚未把 Tensor Engine 跑满。剩下的余量要靠让计算更精简来争取。最清晰的路径是消除浪费,降低产出结果所需的整体 FLOPS。我们围绕模型中的确切形状重写了 attention kernel,以减少花在 padding 上的 FLOPS。这正是 kernel 定制真正见效的地方。通用 kernel 为处理任意形状而构建,因此无法利用特定架构中的具体形状。我们按照模型 head 架构的真实形状来定制实现,从 attention 中榨出了更多性能。我们还把 attention 运算合并为单次 matmul,而通用 kernel 需要多个指令步骤。这减少了计算相同数值所需的内层循环运算,从而减少了 HBM 往返,带来了最后这轮显著的加速。

总体来看,与最初的基线相比,我们的最终版本在每嵌入 100 万份文档的性能和成本上实现了 47% 的提升。

智能体在一个简单的三部分测试框架内开发出上述所有优化,该框架让它们能够验证正确性、测量性能,并对工作负载进行分析以找到下一步方向。借助机器学习运维工作流平台 Flyte,每个智能体运行各自的工作流。该工作流会编译一个 NEFF Neuron 可执行文件,并针对基线数据集检查一次前向传播,其结果必须与规范模型达到 0.999 的余弦相似度。随后它会分析一次前向传播的性能,并在真实硬件上运行 30 分钟的饱和基准测试。

该测试框架以不同成本支持各类智能体。轻量级模型负责网格搜索编译器参数这类繁琐工作,而正确性检查可防止偏移。推理型智能体测试复杂假设并找出瓶颈,而最强的代码生成模型则编写自定义内核来修复这些问题。让每个智能体匹配其任务可控制成本,验证则防止能力强的智能体出现偏移。由于许多智能体不了解最新的特性和语法,neuron 智能体式开发技能也发挥了作用。

到目前为止,我们已针对美国西部(俄勒冈)区域(us-west-2)对 Inferentia2 进行了优化,但在单一区域调优单一硬件系列存在硬性限制。Spot 和按需实例的可用性因区域而异,拥有众多使用方的组织会与自己形成竞争。锁定单一区域或实例类型几乎必然导致算力短缺,并可能造成服务降级。

Exa 通过一个 Kubernetes 控制平面在多个 Region 之间调度算力。我们的调度器会比较各实例类型的每美元性能,把 Pod 打包到最便宜的节点上,并在嵌入需求变化时重新调度。我们每优化一种硬件类型,就更接近全局最优。正因如此,我们在 Inferentia2 上的工作可以直接沿用到 Trainium2。切换硬件配置档让我们能腾出其他团队需要的节点,或吸收他们未使用的预留算力。对于一个长期存续的模型而言,跨硬件优化带来的回报极其巨大。

在 Exa 所处的规模上,关键是从一个灵活的硬件资源池中榨取最大的性价比。把 Neuron 硬件加入这个资源池,并用 NKI 对其进行优化,已被证明是长期成本节约的一大来源。

Summer 是 Exa Labs 的数据基础设施工程师,专注于解决 Web 规模下的杂乱数据问题。

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips)

At Exa, we are building one of the few true web indexes by aggregating and serving knowledge from across the web, along with private and synthetic sources. We optimize the index for semantic search, grounding agents and humans in true, high-quality data. We use multi-tower semantic machine learning (ML) models to embed web data and query data into the same vector space, so users can express complex queries in natural language alongside traditional keyword search. The keystone technology behind this is an in-house vector database. It must ingest as much high-quality web data as possible and embed all of it. The unit economics of embedding therefore underlie everything we do. At the scale of tens or hundreds of billions of documents, finding the hardware that delivers the best performance per dollar isn’t a tuning detail. Any per-document cost improvement compounds into a significant number.

This post details how we optimized our embedding infrastructure for cheaper, more flexible inference workloads on AWS Neuron hardware (AWS Inferentia2 and AWS Trainium2), and how an agentic development workflow drove the optimization.

Inference runs on an asynchronous pipeline, but it must be high-volume and fast to serve recent data quickly and shrink the time between discovery and serving.

Exa stack showing web data flowing from crawlers through parse, moderate, and embed stages into a vector database

Figure 1: A simplified view of the Exa stack

Every structured web document we rank as high enough quality must be embedded before it can be served. This figure shows how raw web data moves from Crawlers into an asynchronous pipeline of Parse, Moderate and filter, then Embed, with the result served from a vector database.

Embedding documents at scale

Our experience with the Neuron hardware family began with a backfill. Moving to a superior document model meant re-embedding our existing corpus. We selected 30 billion high-priority documents for re-embedding. A migration like this temporarily doubles the embedding load, which made it exactly the right moment to re-evaluate our hardware choices.

Two parallel embedding pipelines sharing crawl and parse stages, doubling the inference load during the backfill

Figure 2: The doubled inference load from the backfill

The preceding figure shows two parallel pipelines sharing the crawl and parse stages, one embedding with the live model into the live index and one re-embedding priority documents with the new model into the new index, which is what doubles the load. This doubling was a clear trigger to reconsider our hardware and optimization choices.

Optimization journey

AWS Inferentia2 figures refer to performance on the smallest instance type, an Inferentia2 xlarge instance (inf2.xlarge), with one chip and two NeuronCores. Our workload runs on a single NeuronCore, so we run Kubernetes pods that pack onto larger instances without leaving any cores idle, giving us container homogeneity across the embedding fleet. The same process runs on a Trainium2 NeuronCore and scales the same way through data parallelism. Our initial Inferentia2 implementation was a straightforward data-parallel, FP16 representation of the model. From that baseline, it became a march to push throughput (docs/sec) as high as possible, using the Neuron SDK and support from the AWS Trainium team. The following chart shows the throughput gains at each milestone.

Chart of throughput gains across optimization milestones, rising to about 1.5 times the baseline

Figure 3: Throughput improvement across optimization milestones

Starting from the naive compile, a progressive series of optimizations ultimately delivered roughly a 1.5× throughput improvement. The first was weight layout optimization, which eliminates unnecessary transposes by aligning the weight layout with the scheduling and layout of the broader model. We also enabled autocast, letting the compiler downcast matrix multiplications to lower precision while preserving dynamic range. The downcast outputs met our correctness gate of 0.999 cosine similarity against the reference (see Agentic methodology), so the accuracy impact was within our acceptance threshold.

NKI kernels

The Neuron Kernel Interface (NKI) library provides optimized kernels for common model layers and operations. These pre-built kernels deliver immediate gains on common features, and we used them to quickly remove the most straightforward sources of inefficiency. A drop-in attention kernel gave us the next bump in performance.

Memory layout optimization: A fused SwiGLU MLP kernel

The next bottleneck was memory spill. Profiles from the Neuron Explorer consistently showed large amounts of spill and reload from high-bandwidth memory (HBM), which kept the Tensor Engine from being fully utilized through the forward pass. Using a drop-in MLP kernel as a steppingstone to profile the shortcomings of the pre-built implementation, we wrote a custom weight-stationary fused SwiGLU MLP kernel that eliminated the HBM stalls.

We used the higher-level NKI language for layout operations, loops, and memory operations, and only dropped into the lower-level NKI ISA for specific tensor operations. Keeping most of the kernel in the higher-level language makes it more understandable for engineers, and easier to debug.

FLOPS reduction

At this point we hit a wall. Per-accelerator throughput measured with the other on-chip core idle was significantly higher than what we saw with both cores running simultaneously. That gap meant our theoretical data-parallel throughput should have been well above our realized aggregate performance, which pointed to contention between the cores.

Profiling and experimentation showed that our workload wasn’t yet saturating the Tensor Engine. The remaining headroom was ours to claim by making the computation leaner. The clearest path left was to reduce the overall FLOPS needed to produce our results by eliminating waste. We rewrote the attention kernel around the exact shapes in our model to cut FLOPS spent on padding. This is where kernel customization really paid off. Generic kernels are built to handle any shape, so they can’t take advantage of the specific shapes in a given architecture. We wrung more performance out of attention by tailoring the implementation to the real shape of our model’s head architecture. We also folded attention operations into single matmuls where the generic kernel used multiple instruction steps. That cut the inner-loop operations needed to compute the same values, which reduced HBM round trips and produced the final significant speedup.

In total, our final version was a 47% improvement in performance and cost per 1 million documents embedded, against our initial baseline.

Agentic methodology

Agents developed all of these optimizations inside a simple three part harness that lets them verify correctness, measure performance and profile the workload to find the next direction. Using Flyte, an ML ops workflow platform, each agent runs its own workflow. The workflow compiles a NEFF Neuron executable and checks a forward pass against a baseline dataset, which must reach 0.999 cosine similarity to the canonical model. It then profiles one forward pass and runs a 30 minute saturation benchmark on real hardware.

The harness supports agents at different costs. Lightweight models handle tedious work like grid searching compiler parameters, and correctness checks prevent drift. Reasoning agents test complex hypotheses and find bottlenecks, while the strongest code generators write custom kernels to fix them. Matching each agent to its task keeps costs in check, and validation keeps capable agents from drifting. Because many agents don’t know the latest features and syntax, neuron agentic development skills also helped.

The benefits of flexible compute

So far, we have optimized Inferentia2 for the US West (Oregon) Region (us-west-2), but tuning one hardware family in one Region has hard limits. Spot and on demand availability vary by Region, and organizations with many consumers compete with themselves. Locking into one Region or instance type almost guarantees a compute shortage, and possibly degraded service.

Exa schedules compute across multiple Regions from one Kubernetes control plane. Our scheduler compares performance per dollar across instance types and packs pods onto the cheapest nodes, rescheduling as embedding demand shifts. Each hardware type we optimize for brings us closer to the global optimum. This is why our Inferentia2 work carried over directly to Trainium2. Shifting hardware profiles lets us free up nodes other teams need or absorb their unused reserved compute. For a long lived model, optimizing across hardware pays off enormously.

Conclusion

At the scale Exa operates, extracting maximum cost-performance from a flexible pool of hardware is what matters. Adding Neuron hardware to that pool and optimizing it with NKI has proven to be a major source of long-term cost savings.


About the authors

Summer Devlin

Summer Devlin

Summer is a data infrastructure engineer at Exa Labs, who focuses on solving messy data problems at web scale.

Jim Burtoft

Jim Burtoft

Jim is a Senior Trainium Solutions Architect at AWS and works directly with companies and educational institutions as well as the team at Hugging Face to make it easier to use models on Trainium. Jim is a CISSP, leader of the Neuron Data Science Technical Field Community, and works with the open source community to enable broader use of Neuron and Trainium. Jim holds a bachelor’s degree in mathematics from Carnegie Mellon University and a master’s degree in economics from the University of Virginia.

Pinak Panigrahi

Pinak Panigrahi

Pinak works on performance optimization of AI models on AWS Trainium at Annapurna Labs, Amazon.

Venu Geddam

Venu Geddam

Venu is a Senior Technical Account Manager with AWS Enterprise Support, where he partners with strategic customers to run resilient, high-scale workloads on AWS. His work spans EC2 capacity and storage-optimized compute, cost and operational optimization, and the infrastructure behind modern AI/ML platforms — including GPU and Graviton fleets, Amazon SageMaker HyperPod, and Amazon Bedrock. Venu helps engineering teams turn ambitious technical goals into production-ready, operationally sound architectures.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 AWS Machine Learning(Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗