11-19 更新 实时
首页/ 技术栈/ 正文

用 EC2 Spot 优化 AssemblyAI 语音转文本推理成本

技术栈 来源:AWS Machine Learning 2026-11-14 03:35 0 阅读 机器翻译
用 EC2 Spot 优化 AssemblyAI 语音转文本推理成本

语音转文本推理的计算量非常密集。每个音频文件都会触发一条由深度学习模型(声学模型、语言模型以及后处理阶段)组成的流水线,这些模型运行在 NVIDIA GPU 上以满足延迟要求。在规模化场景下,这意味着数千个 GPU 实例持续运行。

AssemblyAI 需要一种方式,在不牺牲客户所依赖的可用性和延迟保障的前提下,以低成本、规模化地运行 GPU 密集型推理负载。完全依赖按需实例(On-Demand Instances)运行会因成本过高而难以为继。Spot 实例能带来显著的成本节省,但当其他地方需要容量时,Amazon Elastic Compute Cloud(Amazon EC2)会提前 2 分钟发出警告并回收这些实例。AssemblyAI 需要一套架构,能够平稳地吸收 Spot 实例中断,同时维持服务级别协议(SLA)承诺。此外,医疗、金融和法律等受监管行业的客户必须将音频数据保留在特定的地理边界内。他们在欧洲的 AWS 区域实施了严格的数据驻留配置。

AssemblyAI 在 AWS Marketplace 上提供 API,用于转录、说话人检测、情感分析、章节检测以及个人可识别信息(PII)脱敏。其模型为科技、媒体、医疗和金融服务等行业的大规模负载转录并分析语音。

在本文中,我们将展示 AssemblyAI 如何为其语音转文本平台大规模运行成本优化的 AI 推理。该架构采用 Amazon Elastic Container Service(Amazon ECS)、Amazon Elastic Compute Cloud(Amazon EC2)竞价型实例以及 Amazon Simple Queue Service(Amazon SQS)。这些服务共同组合出一套灵活的 NVIDIA GPU 机群、ECS 容量提供程序以及基于队列的重试机制。其结果是:在显著节省成本的同时,仍能在多个欧洲 AWS 区域履行 SLA 承诺。

AssemblyAI 的推理平台运行在 Amazon ECS 上,计算容量通过 ECS 容量提供程序进行管理。该机群在多个 NVIDIA GPU 实例族中混用 Amazon EC2 竞价型实例与按需实例,每块 GPU 运行单个推理模型,规模达数千个实例。该架构建立在四大支柱之上,它们协同工作,共同实现成本优化且具备韧性的推理:

下图展示了该方案的部署方式:

图 1:AssemblyAI 在 Amazon ECS 上的推理架构,采用 Amazon EC2 竞价型 GPU 实例以及基于 Amazon SQS 的重试机制

一个 Amazon ECS 集群运行在跨多个可用区的虚拟私有云(VPC)中。该集群承载由 GPU 支持的 ECS 任务(图中显示为 ECS Task, GPU Spot 1 和 ECS Task GPU Spot 2),这些任务轮询队列并处理推理作业。每个任务从 Amazon Simple Storage Service(Amazon S3)获取音频文件,并将完成的转写结果写入结果存储,AssemblyAI 通过配置该存储将输出交付给终端客户。

当 Spot 实例收到中断通知时,Amazon EventBridge 会调用 Spot 中断处理器(一个 AWS Lambda 函数)。该处理器把正在执行中的任务送回主任务 SQS 队列(Main Job SQS Queue),以便另一个健康的 ECS 任务能够接手处理。客户可将系统配置为把耗尽重试次数的任务路由到死信 SQS 队列(Dead Letter SQS Queue)以便排查。Amazon CloudWatch Metrics 监控整个机群的健康状况与 Spot 中断事件,为整个系统提供可观测性。

为了在不牺牲可靠性的前提下优化推理成本,AssemblyAI 在每个 ECS 集群内的同一个容量提供商策略中配置了多个容量提供商。按需容量提供商作为始终在线的基线,而 Spot 容量提供商则横跨不同的 GPU 实例族和可用区,用于承接需求峰值。每个容量提供商都与一个 Amazon EC2 Auto Scaling 组关联。Amazon ECS 托管扩缩容会根据容量提供商权重和待处理的推理任务,自动对 GPU 实例进行扩缩容。当任务队列增长时,Spot 容量提供商从多样化的资源池中申请更多实例;当需求下降时,实例则平滑缩容。

Spot 最佳实践中最有效的一条,是在广泛的实例类型上保持灵活性,以提高获得所需总算力的机会。AssemblyAI 的基础设施并不依赖单一的 GPU 实例族,而是横跨三个 Amazon EC2 GPU 实例族:

通过将需求分散到不同的实例系列和规格上,AssemblyAI 能够从规模大得多的可用 Spot 容量池中获取资源。借助价格-容量优化分配策略,Amazon EC2 会从可用容量最高、价格最低的 Spot 容量池中分配实例,从而降低中断发生的可能性。

在 Spot 中断期间维持服务质量是一项核心设计要求。AssemblyAI 使用 Amazon SQS 将任务提交与执行解耦。当转录请求到达时,它会被放入 Amazon SQS 队列。运行在 GPU 实例上的 Amazon ECS 任务轮询该队列,拉取任务并执行推理流水线。单个任务通常需要 15 秒到数分钟不等,具体取决于音频长度。

当 Spot 实例收到 2 分钟的中断通知时,重试流程如下:

为缩短恢复时间,AssemblyAI 创建了包含模型产物和容器镜像的自定义 Amazon Machine Image(AMI),耗时约 2 分钟。对终端客户而言,Spot 中断的影响是处理时间略有增加,但转录仍能成功完成并按预期交付。

在多个 AWS Region 中运行,使 AssemblyAI 能够服务受监管行业的客户,包括医疗健康、金融服务和法律服务,在这些领域,音频数据绝不能离开特定的地理边界。每个 AWS Region 都运行一套完全独立的推理架构副本:各自拥有 Amazon SQS 队列、Amazon ECS 集群、容量提供者和 GPU 资源池。这些欧洲 Region 中可用的 GPU 实例规格族有多个(g4dn、g5、g6),且 Spot 价格具有竞争力。这种实例多样性支撑了可靠的基于 Spot 的推理服务,同时满足严格的数据驻留要求。

通过在生产环境中运行搭载 GPU 的 Amazon EC2 Spot 实例,AssemblyAI 在推理算力上相比按需定价实现了 70% 的成本节省。随着平台的计算需求同比增长近 4 倍,基于 Spot 的架构使单位经济效益保持可持续。在这一规模下运营,AssemblyAI 认识到,在实例规格族和规格大小之间做多元化,是降低 Spot 中断率最有效的手段。g4dn、g5 和 g6 规格族接入的是彼此独立的容量池,因此当某个规格族出现容量压力时,其他规格族仍然可用。从第一天起就通过基于队列的重试和快速预热来设计容错能力,使 Spot 中断变成常规事件,同时仍能履行 SLA 承诺。

运行具有类似特征的批处理或近实时 AI 工作负载(在 GPU 算力上运行、生命周期短、可独立重试的任务)的团队,可以套用这一模式。先从实例规格族多元化入手,并构建基于队列的容错能力,从而在不牺牲可用性的前提下,利用高性价比的 Spot 基础设施。

要开始在 Amazon ECS 上使用 GPU 任务,请参阅 Amazon ECS for GPU workloads。若要评估工作负载可用的 Spot 容量,可探索面向加速计算的 Spot 放置评分以及 Amazon EC2 Spot 实例最佳实践。AssemblyAI 的 Speech-to-Text API 已在 AWS Marketplace 上架。如需进一步了解其平台,请访问 assemblyai.com。

Belair 是 AWS 的 ISV 技术客户经理,专注于容器与无服务器计算。他帮助企业客户优化其在 Amazon ECS 和 EKS 上的工作负载。业余时间,Belair 喜欢旅行和户外活动。

Sandeep 是 AWS 的 ISV 解决方案架构师,与 AI/ML 领域的 ISV 客户合作,在 AWS 上设计可扩展且高性价比的解决方案。他专长于帮助客户大规模部署生成式 AI 和机器学习工作负载。工作之余,Sandeep 喜欢关注 AI 研究的最新进展。

Mimi 是 AssemblyAI 的软件工程师,负责构建和维护支撑大规模语音转文字推理的基础设施。她主要关注 GPU 调度、容器编排和成本优化。工作之余,Mimi 喜欢享受完整而不被打扰的周末。

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Cost-optimizing AssemblyAI speech-to-text inference on AWS with Amazon EC2 Spot)

Speech-to-text inference is computationally intensive. Each audio file triggers a pipeline of deep learning models (acoustic models, language models, and post-processing stages) running on NVIDIA GPUs to meet latency requirements. At scale, this means thousands of GPU instances running continuously.

AssemblyAI needed a way to run GPU-intensive inference workloads cost-effectively at scale, without sacrificing the availability and latency guarantees their customers depend on. Running entirely on On-Demand Instances would be cost-prohibitive. Spot Instances offer significant savings, while Amazon Elastic Compute Cloud (Amazon EC2) can reclaim them with a 2-minute warning when capacity is needed elsewhere. AssemblyAI needed an architecture that absorbs Spot interruptions gracefully while maintaining service level agreement (SLA) commitments. Additionally, customers in regulated industries, including healthcare, finance, and legal, must keep audio data within specific geographic boundaries. They implement strict data residency configurations in European AWS Regions.

Available on AWS Marketplace, AssemblyAI provides an API for transcription, speaker detection, sentiment analysis, chapter detection, and personally identifiable information (PII) redaction. Their models transcribe and analyze speech for high-volume workloads across industries including technology, media, healthcare, and financial services.

In this post, we show you how AssemblyAI runs cost-optimized AI inference at scale for their speech-to-text platform. The architecture uses Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, and Amazon Simple Queue Service (Amazon SQS). Together, these services combine a flexible fleet of NVIDIA GPUs, ECS capacity providers, and a queue-based retry mechanism. The result is significant cost savings while maintaining SLA commitments across European AWS Regions.

Solution overview

AssemblyAI’s inference platform runs on Amazon ECS with compute capacity managed through ECS capacity providers. The fleet mixes Amazon EC2 Spot Instances and On-Demand Instances across multiple NVIDIA GPU instance families, running a single inference model per GPU at a scale of thousands of instances. The architecture is built on four pillars that work together to deliver cost-optimized and resilient inference:

  1. Amazon ECS capacity providers mixing Spot and On-Demand.
  2. Multi-family GPU Spot diversification across EC2 instance types (such as g4dn.xlarge, g4dn.2xlarge, g5.xlarge, g5.4xlarge, and g6.2xlarge).
  3. Queue-based retry architecture using Amazon SQS for interruption recovery.
  4. Regional deployment across US West (Oregon) us-west-2 and Europe (London) eu-west-2 for data residency compliance.

The following diagram illustrates how this solution is deployed:

AssemblyAI inference architecture: Amazon ECS GPU Spot tasks poll an Amazon SQS queue, an AWS Lambda function returns interrupted jobs, and Amazon CloudWatch monitors the fleet

Figure 1: AssemblyAI’s inference architecture on Amazon ECS with Amazon EC2 Spot GPU instances and an Amazon SQS-based retry mechanism

An Amazon ECS cluster runs within a virtual private cloud (VPC) across multiple Availability Zones. The cluster hosts GPU-backed ECS tasks (shown as ECS Task, GPU Spot 1 and ECS Task GPU Spot 2) that poll the queue and process inference jobs. Each task retrieves audio files from Amazon Simple Storage Service (Amazon S3) and writes the completed transcription to a results store, which AssemblyAI configures to deliver the output to the end customer.

When a Spot Instance receives an interruption notice, Amazon EventBridge invokes the Spot Interruption Handler (an AWS Lambda function). The handler returns the in-flight job to the Main Job SQS Queue so another healthy ECS task can pick it up. Customers configure their system to route jobs that exhaust retry attempts to a Dead Letter SQS Queue for investigation. Amazon CloudWatch Metrics monitors the health of the fleet and the Spot interruption events, providing observability across the system.

Amazon ECS capacity providers and Spot diversification

To optimize inference costs without compromising reliability, AssemblyAI configures multiple capacity providers in a single capacity provider strategy within each ECS cluster. An On-Demand capacity provider serves as an always-on baseline, and a Spot capacity provider spans different GPU instance families and Availability Zones to serve spikes of demand. Each capacity provider associates with an Amazon EC2 Auto Scaling group. Amazon ECS managed scaling automates the scaling of GPU instances based on the capacity provider weights and the pending inference tasks. When the task queue grows, the Spot capacity provider requests additional instances from the diversified pool. When demand drops, instances scale in gracefully.

One of the most effective Spot best practices is to be flexible across a wide range of instance types to increase the chances of getting the aggregate compute capacity needed. Rather than depending on a single GPU instance family, AssemblyAI’s infrastructure spans three Amazon EC2 GPU instance families:

  1. g4dn instances – Powered by NVIDIA T4 GPUs, providing a cost-effective baseline for standard speech recognition models.
  2. g5 instances – Equipped with NVIDIA A10G GPUs, delivering higher throughput for more demanding model architectures.
  3. g6 instances – Featuring NVIDIA L4 GPUs, offering improved performance-per-dollar for larger models including large language model-based inference.

By spreading demand across instance families and sizes, AssemblyAI draws from a significantly larger pool of available Spot capacity. Using the price-capacity optimized allocation strategy, Amazon EC2 provisions instances from the Spot capacity pools with the highest available capacity at the lowest price, reducing the likelihood of interruptions.

Queue-based retry for interruption resilience

Maintaining quality of service during Spot interruptions was a core design requirement. AssemblyAI decouples job submission from execution using Amazon SQS. When a transcription request arrives, it’s placed into an Amazon SQS queue. Amazon ECS tasks running on GPU instances poll the queue, pull jobs, and execute the inference pipeline. Individual jobs typically take between 15 seconds and several minutes depending on audio length.

When a Spot Instance receives a 2-minute interruption notice, the retry flow works as follows:

  1. In-flight jobs that can complete within the 2-minute window finish normally. Jobs that need more time are returned to the Amazon SQS queue, where they become immediately available for other workers.
  2. Another healthy GPU instance picks up the message, potentially on a different instance type or in a different Availability Zone, and completes the inference.

To reduce recovery time, AssemblyAI creates custom Amazon Machine Images (AMIs) containing model artifacts and container images in approximately 2 minutes. For end customers, the impact of a Spot interruption is a small increase in processing time, but the transcription completes successfully and is delivered as expected.

Regional deployment for data residency

Running in multiple AWS Regions allows AssemblyAI to support customers in regulated industries, including healthcare, financial services, and legal services, where audio data must never leave specific geographic boundaries. Each AWS Region runs a fully independent copy of the inference architecture: its own Amazon SQS queues, Amazon ECS clusters, capacity providers, and GPU fleets. Multiple GPU instance families (g4dn, g5, g6) are available in these European Regions with competitive Spot pricing. This instance diversity supports a reliable Spot-based inference service while meeting strict data residency requirements.

Results

By running Amazon EC2 Spot Instances with GPUs in production, AssemblyAI achieved 70 percent cost savings on inference compute compared to On-Demand pricing. As the platform grew nearly 4x year-over-year in compute demand, the Spot-based architecture kept unit economics sustainable. Operating at this scale, AssemblyAI learned that diversifying across instance families and sizes is the most effective lever for minimizing Spot interruption rates. The g4dn, g5, and g6 families tap into independent capacity pools, so when one family experiences capacity pressure, the others remain available. Designing fault tolerance from day one with queue-based retries and fast warm-up turns Spot interruptions into routine events while maintaining SLA commitments.

Conclusion

Teams running batch or near real-time AI workloads with similar characteristics (short-lived, independently retriable jobs on GPU compute) can apply this pattern. Start by diversifying instance families and building queue-based fault tolerance to take advantage of cost-effective Spot infrastructure without compromising availability.

To get started with GPU tasks on Amazon ECS, see Amazon ECS for GPU workloads. To evaluate Spot availability for your workload, explore using Spot placement scores for accelerated compute and Amazon EC2 Spot Instances best practices. AssemblyAI’s Speech-to-Text API is available on AWS Marketplace. To learn more about their platform, visit assemblyai.com.


About the authors

Belair Albahagwi

Belair Albahagwi

Belair is a Technical Account Manager – ISV at AWS, specializing in containers and serverless compute. He helps enterprise customers optimize their workloads on Amazon ECS and EKS. In his spare time, Belair enjoys traveling and outdoor activities.

Sandeep Kampa

Sandeep Kampa

Sandeep is a Solutions Architect – ISV at AWS, where he works with AI/ML ISV customers to architect scalable, cost-effective solutions on AWS. He specializes in helping customers deploy generative AI and machine learning workloads at scale. Outside of work, Sandeep enjoys exploring the latest developments in AI research.

Mimi Chiang

Mimi Chiang

Mimi is a Software Engineer at AssemblyAI, where she builds and maintains the infrastructure powering speech-to-text inference at scale. She focuses on GPU scheduling, container orchestration, and cost optimization. Outside of work, Mimi enjoys proper weekends.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 AWS Machine Learning(Cost-optimizing AssemblyAI speech-to-text inference on AWS with Amazon EC2 Spot),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗