11-19 更新 实时
首页/ 算力/ 正文

SageMaker AI 双向流式推理推出按用量计费

算力 来源:AWS Machine Learning 2026-11-11 23:10 0 阅读 机器翻译
SageMaker AI 双向流式推理推出按用量计费

针对双向流式传输的按用量计费,解决了实时语音 AI 的软件计费问题。在 Amazon SageMaker AI 双向流式传输上部署流式模型的客户,需为承载该模型的端点基础设施付费。另外,在 AWS Marketplace 上架模型的提供商也需要一种方式,对客户在实时流中实际消耗的部分进行收费。今天,我们宣布推出 Amazon SageMaker 双向(bidi)流式推理的按用量计费,让模型提供商能够为 AWS Marketplace 的软件费用上报实际用量。Amazon SageMaker AI 端点基础设施费用仍然适用。

实时语音 AI 公司 Deepgram 是这一能力的首发合作伙伴,将按用量的语音转文本(STT)和文本转语音(TTS)计费引入 AWS Marketplace。本文延续了 Deepgram 与 AWS 之间的一系列创新。首先,我们推出了 Amazon SageMaker AI 上用于实时推理的双向流式传输,支持客户端与模型服务器之间的长连接双向流。随后,Deepgram 通过 AWS Identity and Access Management(AWS IAM)临时委派增强了对 Amazon SageMaker AI 的支持,简化了企业部署的安全认证,并通过增强指标扩展了可观测性。如今,借助按用量计费,我们进一步夯实了生产级实时 AI 的基础,使计费方式与客户消费流式和实时 AI 工作负载的方式保持一致。

借助针对 SageMaker AI 实时请求-响应推理的 AWS Marketplace 软件计量,模型提供商可以在 HTTP 响应头中上报单位数量。AWS Marketplace 使用该数量来计算软件费用。这适用于请求-响应式 API,即在发送响应时即可确定计费数量的场景。

双向流式传输则不同。客户端通过 HTTP/2 事件流,经由 SageMaker AI API 流式传输音频并接收转写结果。SageMaker AI 通过 WebSocket 连接到模型服务器。一个会话可能持续 30 秒,也可能持续 30 分钟。用量会随会话运行而不断累积,直到会话结束才能得知最终总量。仅靠一个响应头,无法在这段实时会话期间报告周期性的用量。

按小时计费的软件定价模式只要模型处于可用状态就会向客户收费,包括空闲时段。而采用基于用量的计量方式,提供商可以让 AWS Marketplace 的软件费用与上报的用量保持一致。SageMaker AI 端点基础设施费用仍单独计算,并在端点预置期间持续计费。

该方案在现有的推理流之外引入了第二条 WebSocket 连接。当双向流式会话启动时,SageMaker AI 会与模型服务器建立两条连接:

模型服务器通过在 WebSocket 升级响应中返回一个标头来选择启用该功能:

X-Amzn-SageMaker-Metadata-Stream-Supported: true

随后,SageMaker AI 会在 /invocations-bidirectional-stream-metadata 路径上向模型服务器打开一条配套 WebSocket。两条连接都携带相同的 X-Amzn-SageMaker-Request-Id 标头,模型服务器借助它把计量记录与当前活跃的推理会话关联起来。

模型服务器在元数据连接上以 JSON 文本帧的形式发送计量数据。发送记录的频率应与计费模式相匹配:随着用量累积增量发送(推荐),既可以按每个单位被消耗时发送,也可以按自上次记录以来已消耗的单位定期批量发送。会话一结束,就立即发送任何最终记录。每个帧只包含一条计量记录,而不是一个记录数组。元数据流的速率配额为每条连接每秒一条消息,最大消息大小为 512 字节。

我们建议随着用量累积增量发送计量记录。这样可以减少会话意外结束时未上报的用量。仅完成传输投递并不能说明用量已完成计费。请将上报的单位数与 AWS Marketplace 报告以及买家扣费进行核对。

{ "Metering": { "Dimension": "inference.count", "ConsumedUnits": 720, "ClientToken": "a3f2b1c0-unique-uuid" } } 计量消息使用 inference.count 维度,ConsumedUnits 提供该维度的数量。如果产品按音频秒数计费,那么一条上报 12 分钟计量音频的记录会以 720 秒作为 ConsumedUnits,即 ConsumedUnits: 720。维度名并不意味着每个请求一个单位。由提供商定义哪些音频算作计量用量,AWS Marketplace 商品再将该单位映射到其价格。商品信息显示每个单位一个价格,标注为按请求计费,最多保留三位小数。ClientToken 字段支持幂等性。如果同一连接上收到重复的 Token,SageMaker AI 会静默丢弃这条重复消息。

元数据连接基于 TCP 传输,TCP 在传输层提供有序、可靠的投递。如果连接建立失败或在会话中途断开,默认情况下推理会继续,因为 X-Amzn-SageMaker-Metadata-Stream-Required 默认值为 false。当提供方要求元数据流不可用时必须停止推理,可将该标头设为 true。这一选择是以推理可用性换取计量连续性。它并不保证格式错误或丢失的记录会被计费。

如果你为转录、联络中心、语音智能体或字幕生成部署实时 AI 模型,新的计量模式会改变 AWS Marketplace 软件费用的计算方式。SageMaker AI 端点基础设施费用仍然照常收取。

通过 AWS Marketplace 实现的透明按单位计费,意味着用量会直接计入你的 AWS 账单。你可以获得合并的 AWS 账单、通过私有报价协商的费率,且无需再管理与模型提供方之间的单独计费关系。

按照产品的单位定义,为已处理的音频付费。如果产品按音频秒数计费,12 分钟的计量音频等于 720 个计费秒,ConsumedUnits: 720。连接保持打开 12 分钟本身并不界定可计费的音频数量。计量用量由提供方的计数规则决定,而端点基础设施费用仍单独计算。

模型提供方可以通过 AWS Marketplace 私有报价提供协商的按单位费率或固定价格合同。

我们建议直接与你的 AWS 客户代表或合作伙伴代表沟通。

如果你是上架 AWS Marketplace 的模型提供方,可以将模型服务器配置为在流式推理会话期间通过伴生元数据流上报用量。

SageMaker Marketplace Onboarding Agent Skill 通过一条可选的推理计量路径来加速接入流程。其入门模板声明支持元数据流,并为配套的 WebSocket 构建经过验证的增量 ConsumedUnits 帧。提交前请审查代码并完成生产级加固。

通过定义一单位代表什么,并在会话过程中上报单位数,使计费与你的定价模型保持一致。AWS Marketplace 对实时推理只收取单一的按单位价格。

以下三步操作说明介绍了如何集成你的模型服务器,以便在双向流式推理会话期间上报计量数据。

当 SageMaker AI 向你的模型服务器发起数据 WebSocket 连接时,请在 WebSocket 升级(101)响应中包含以下标头:

你的模型服务器返回 X-Amzn-SageMaker-Metadata-Stream-Supported: true 后,SageMaker AI 会在默认路径 /invocations-bidirectional-stream-metadata(或你通过 X-Amzn-SageMaker-Metadata-Invocation-Path 指定的自定义路径)上向你的模型服务器发起第二条 WebSocket 连接。

SageMaker AI 会在数据 WebSocket 和元数据 WebSocket 的握手中都包含 X-Amzn-SageMaker-Request-Id 标头。你的模型服务器必须执行以下操作:

在元数据连接上,以 WebSocket 文本帧中的 JSON 对象形式发送计量数据。每一帧必须恰好包含一条计量记录。在本示例中,该产品按音频秒数计费,该记录将 12 分钟的计量音频上报为 720 个单位:

{ "Metering": { "Dimension": "inference.count", "ConsumedUnits": 720, "ClientToken": "a3f2b1c0-unique-uuid" } } 消息字段:

随着用量累积,增量发送计量记录,以减少会话意外结束时出现未上报用量的情况。将上报的数量与 AWS Marketplace 报告及买家账单进行对账,以核实计费,其中包括中断会话的测试。

Required 请求头控制元数据连接不可用时会发生什么。将其设为 true 可停止推理,或使用默认值 false 让推理以仅数据模式继续运行。这是在计量连续性与推理可用性之间做出的取舍。无论采用哪种设置,格式错误的消息都可以被丢弃。

使用 Limited 上架和加入允许列表的测试买家来验证集成。使用合适的报价(必要时包括私有报价),以便测试覆盖预期的单位和价格映射。

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Introducing usage-based pricing for Amazon SageMaker AI bidirectional streaming)

Usage-based metering for bidirectional streaming solves a software billing problem for real-time speech AI. Customers deploying streaming models on Amazon SageMaker AI bidirectional streaming pay for the endpoint infrastructure that hosts the model. Separately, model providers listing on AWS Marketplace need a way to charge for what customers consume during a live stream. Today, we’re announcing usage-based metering for Amazon SageMaker bidirectional (bidi) streaming inference, so model providers can report actual usage for the AWS Marketplace software charge. Amazon SageMaker AI endpoint infrastructure charges still apply.

Deepgram, the real-time speech AI company, is the launch partner for this capability, bringing usage-based Speech-to-Text (STT) and Text-to-Speech (TTS) billing to AWS Marketplace. This post builds on a series of innovations between Deepgram and AWS. First, we introduced bidirectional streaming for real-time inference on Amazon SageMaker AI, which supports long-lived bidirectional streams between clients and model servers. Then, Deepgram enhanced Amazon SageMaker AI support with AWS Identity and Access Management (AWS IAM) temporary delegation, simplifying secure authentication for enterprise deployments, and expanded observability with enhanced metrics. Now, with usage-based metering, we further strengthen the foundation for production real-time AI, aligning billing with how customers consume streaming and real-time AI workloads.

Why streaming broke the existing billing model

With AWS Marketplace software metering for SageMaker AI real-time request-response inference, model providers can report a unit count in an HTTP response header. AWS Marketplace uses that quantity for the software charge. This works for request-response APIs where the billable quantity is known when the response is sent.

Bidirectional streaming is different. A client streams audio and receives transcriptions through the SageMaker AI API using an HTTP/2 event stream. SageMaker AI connects to the model server over WebSockets. A session can last 30 seconds or 30 minutes. Usage accumulates as the session runs, and the final total isn’t known until it ends. A response header alone can’t report periodic usage during that live session.

Hourly software pricing can charge customers while a model is available, including quiet periods. With usage-based metering, providers can align the AWS Marketplace software charge with reported usage. SageMaker AI endpoint infrastructure charges remain separate and continue to apply while the endpoint is provisioned.

How companion metadata streaming works

The solution introduces a second WebSocket connection alongside the existing inference stream. When a bidirectional streaming session starts, SageMaker AI establishes two connections to the model server:

  • Data connection: carries inference request and response data, the existing bidirectional stream.
  • Metadata connection: a companion WebSocket dedicated to metering data.

Model servers opt in by returning one header on the WebSocket upgrade response:

X-Amzn-SageMaker-Metadata-Stream-Supported: true

SageMaker AI then opens a companion WebSocket to the model server at /invocations-bidirectional-stream-metadata. Both connections carry the same X-Amzn-SageMaker-Request-Id header, which the model server uses to correlate metering records with the active inference session.

The model server sends metering data as JSON text frames on the metadata connection. Send records at a frequency that matches the billing model: incrementally as usage accrues (recommended), either as each unit is consumed or as periodic batches of units consumed since the previous record. Send any final record as soon as the session ends. Each frame contains exactly one metering record, rather than an array of records. The metadata stream has a rate quota of one message per second per connection, with a maximum message size of 512 bytes.

We recommend sending metering records incrementally as usage accrues. This reduces the amount of usage left unreported if a session ends unexpectedly. Transport delivery alone does not establish that usage has been billed. Validate reported units against AWS Marketplace reports and buyer charges.

{
  "Metering": {
    "Dimension": "inference.count",
    "ConsumedUnits": 720,
    "ClientToken": "a3f2b1c0-unique-uuid"
  }
}

Metering messages use the inference.count dimension, and ConsumedUnits supplies the quantity for that dimension. If the product charges per second of audio, a record reporting 12 minutes of metered audio carries 720 seconds as ConsumedUnits: 720. The dimension name doesn’t mean one unit per request. The provider defines which audio counts as metered usage, and the AWS Marketplace offer maps the unit to its price. The listing shows one price per unit, labeled per request, with up to three decimal places. The ClientToken field supports idempotency. If a duplicate token arrives on the same connection, SageMaker AI drops the duplicate message silently.

The metadata connection runs over TCP, which provides ordered, reliable delivery at the transport layer. If it fails to establish or drops mid-session, inference continues by default because X-Amzn-SageMaker-Metadata-Stream-Required defaults to false. Set this header to true when the provider must stop inference if the metadata stream becomes unavailable. This choice trades inference availability for metering continuity. It doesn’t guarantee that malformed or dropped records are billed.

What this means for customers

If you deploy real-time AI models for transcription, contact centers, voice agents, or captioning, the new metering model changes how the AWS Marketplace software charge is calculated. SageMaker AI endpoint infrastructure charges still apply.

Transparent, per-unit billing through AWS Marketplace means usage flows directly into your AWS bill. You get consolidated AWS billing, negotiated rates through private offers, and no separate billing relationship with the model provider to manage.

Pay for audio processed according to the product’s unit definition. If the product charges per second of audio, 12 minutes of metered audio equals 720 billable seconds and ConsumedUnits: 720. An open connection lasting 12 minutes doesn’t, by itself, define the billable audio quantity. The provider’s counting rules determine the metered usage, whereas endpoint infrastructure charges remain separate.

Model providers can offer negotiated per-unit rates or fixed-price contracts through AWS Marketplace private offers.

We recommend discussing directly with your AWS account representative or partner representative.

What this means for model providers

If you’re a model provider listing on AWS Marketplace, you can configure your model server to report usage during streaming inference sessions through companion metadata streaming.

The SageMaker Marketplace Onboarding Agent Skill accelerates onboarding with an opt-in inference metering path. Its starter templates advertise metadata-stream support and build validated, incremental ConsumedUnits frames for the companion WebSocket. Review and production-harden the code before submission.

Align billing with your pricing model by defining what one unit represents and reporting units as the session progresses. AWS Marketplace charges one per-unit price for real-time inference.

The following three-step walkthrough describes how to integrate your model server to report metering data during bidirectional streaming inference sessions.

Step 1: Opt in using data WebSocket response headers

When SageMaker AI opens the data WebSocket connection to your model server, include these headers in your WebSocket upgrade (101) response:

 

Header Required Values Description
1 X-Amzn-SageMaker-Metadata-Stream-Supported Yes (to opt in) TRUE Signals that your model server supports the companion metadata stream. If absent or not true, no metadata connection is attempted. Without the metadata stream, the session is not metered for the model provider’s software charge.
2 X-Amzn-SageMaker-Metadata-Stream-Required No true / false This header only takes effect when X-Amzn-SageMaker-Metadata-Stream-Supported is true. If true, inference stops when the metadata connection fails. If false or absent, inference continues in data-only mode. Default: false. Choose true when metering continuity takes priority over inference availability. This setting doesn’t guarantee billing for malformed or dropped records.
3 X-Amzn-SageMaker-Metadata-Invocation-Path No Custom path string Overrides the default metadata WebSocket path. Default: /invocations-bidirectional-stream-metadata.

Step 2: Accept the metadata WebSocket connection

After your model server returns X-Amzn-SageMaker-Metadata-Stream-Supported: true, SageMaker AI opens a second WebSocket connection to your model server at the default path /invocations-bidirectional-stream-metadata (or the custom path you specified by using X-Amzn-SageMaker-Metadata-Invocation-Path).

SageMaker AI includes the header X-Amzn-SageMaker-Request-Id on both the data and metadata WebSocket handshakes. Your model server must do the following:

  • Accept the WebSocket upgrade on the metadata path.
  • Use X-Amzn-SageMaker-Request-Id to correlate the metadata connection with the corresponding data connection for the same inference session.

Step 3: Send metering messages

Send metering data as a JSON object in a WebSocket text frame on the metadata connection. Each frame must contain exactly one metering record. In this example, the product charges per second of audio, and the record reports 12 minutes of metered audio as 720 units:

{
  "Metering": {
    "Dimension": "inference.count",
    "ConsumedUnits": 720,
    "ClientToken": "a3f2b1c0-unique-uuid"
  }
}

Message fields:

 

Field Required Type Constraints Description
1 Metering Yes Object — Container for the metering record.
2 Metering.Dimension Yes String Must be inference.count The billing dimension. Must be inference.count.
3 Metering.ConsumedUnits Yes Number 0 or greater The units consumed since your previous record. Each record is billed separately. A value of 0 is accepted but not billed.
4 Metering.ClientToken No String Max 64 characters A unique token (for example, a UUID) for idempotency. If the same ClientToken is sent multiple times on the same connection, only the first occurrence is recorded.

Send metering records incrementally as usage accrues to reduce unreported usage if a session ends unexpectedly. Reconcile the reported quantities with AWS Marketplace reports and buyer charges to verify billing, including interrupted-session tests.

Constraints and limits

 

Limit Value Notes
1 Maximum message size 512 bytes Total size of the JSON text frame
2 Maximum send rate 1 message per second per connection Exceeding this rate might result in dropped messages
3 Maximum Dimension length 128 characters —
4 Maximum ClientToken length 64 characters —
5 Messages per frame Exactly 1 Don’t batch multiple metering records in a single frame
6 Frame type Text frames only Binary frames are not supported for metering

Error handling and failure modes

The Required header controls what happens when the metadata connection becomes unavailable. Set it to true to stop inference, or use the default false to let inference continue in data-only mode. This is a choice between metering continuity and inference availability. Malformed messages can still be dropped with either setting.

 

Scenario Required = true Required = false (default)
1 Metadata WebSocket fails to connect Data stream is terminated Data stream continues normally (data-only mode)
2 Metadata WebSocket drops mid-session Data stream is terminated Data stream continues normally. Metering stops
3 Malformed metering message Message is dropped. Metadata connection stays open Message is dropped. Metadata connection stays open
4 Data connection breaks mid-stream Metadata stream is closed Metadata stream is closed

List your model on AWS Marketplace

  1. Register your AWS account as an AWS Marketplace seller. For paid products, complete the tax and banking information.
  2. Create a SageMaker AI model package from your container in a supported publishing Region, and an AWS Identity and Access Management (IAM) role that AWS Marketplace can assume to read it.
  3. In AWS Partner Central, choose Build, Machine learning, then Create machine learning product, and add the model package as the first version.
  4. Choose per-inference pricing. You set one per-unit price for real-time inference across all instance types, shown per request with up to three decimal places. You also set an hourly price for each batch transform instance type. Choose the pricing model when you create the listing because you can’t switch a released listing between hourly and per-inference pricing.
  5. After validation, request Public visibility. For enterprise buyers, extend private offers with a negotiated per-unit rate or a fixed-price contract.

Validate billing before release

Validate the integration with a Limited listing and an allowlisted test buyer. Use an appropriate offer, including a private offer where needed, so the test exercises the intended unit and price mapping.

  1. Run a known-quantity test. For a product priced per second of audio, send 12 minutes of metered audio and check that the reported records total 720 units.
  2. Reconcile the model-server records with the seller’s usage reports and the buyer’s charges. Allow time for reports to populate and verify the usage type <region code>-SMMP:inference.count.m.i.c (for example, USE1-SMMP:inference.count.m.i.c in us-east-1). Confirm the offer’s unit and price mapping rather than assuming a fixed reporting delay.
  3. Test normal completion, interrupted sessions, metadata-stream loss, and malformed messages. Compare reported and billed quantities for each case with the SageMaker AI and AWS Marketplace teams before release.
  4. Delete test endpoints when testing is complete to avoid ongoing infrastructure charges.

Getting started with Deepgram on AWS Marketplace

To deploy Deepgram with usage-based metering on SageMaker AI, follow these steps.

  • Subscribe to the Deepgram Voice AI streaming model you need on AWS Marketplace, such as Deepgram Nova-3 Monolingual Speech-to-Text (STT) Streaming.
  • Deploy a SageMaker AI real-time endpoint using the Deepgram model package from your AWS Marketplace subscription.
  • Connect to the endpoint using the SageMaker AI bidirectional streaming API and stream audio through its HTTP/2 event stream.

Deepgram’s model server calculates metering automatically, and Amazon SageMaker AI and AWS Marketplace process it. You don’t need to configure anything on the client side.

Clean up

When you finish testing, delete the SageMaker AI real-time endpoint and model resource to stop infrastructure charges. If you no longer need the product, remove the AWS Marketplace subscription.

Conclusion

With usage-based metering for bidirectional streaming, model providers can align AWS Marketplace software pricing with reported usage during a live session. Customers pay the software charge for the metered units defined by the product and offer. Amazon SageMaker AI endpoint infrastructure charges remain separate and still apply. Learn more about Amazon SageMaker AI.

To get started:

Acknowledgements

Special thanks to Kareem Syed-Mohammed (Principal Product Manager), Susmitha Marupaka (Product Marketing Manager), Zhengfei Ji (Software Development Engineer for Amazon SageMaker AI), and EJ Ferrell for reviewing this post and leading the development of this capability.


About the authors

Victor Wang

Victor Wang

Victor is a Staff Software Engineer at Deepgram and Technical Advisor to the VP of Engineering based in San Francisco, CA. Before joining Deepgram, he was a Sr. Solutions Architect at AWS working with generative AI startups and held engineering, consulting, and program management roles across AWS. He enjoys learning new technologies and traveling, having flown over two million miles.

Lingran Xia

Lingran Xia

Lingran is a software development engineer at AWS. He currently focuses on improving inference performance of machine learning models. In his free time, he enjoys traveling and skiing.

Andre Gomes

Andre Gomes

Andre is a Frontier AI Solutions Architect at AWS, working with AI startups on large-scale training, real-time inference, and Amazon SageMaker AI. Before joining AWS, Andre earned his PhD at Universidade Federal de Minas Gerais, with research conducted at the Institute for Manufacturing, University of Cambridge, applying large language models and neural topic modeling to strategic roadmapping.

Daniel Wirjo

Daniel Wirjo

Daniel is a Solutions Architect at AWS, focused on frontier AI startups. As a former startup CTO, he enjoys collaborating with founders and engineering leaders to drive growth and innovation on AWS. Outside of work, Daniel enjoys taking walks with a coffee in hand, appreciating nature, and learning new ideas.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 AWS Machine Learning(Introducing usage-based pricing for Amazon SageMaker AI bidirectional streaming),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗