11-19 更新 实时
首页/ 算力/ 正文

Amazon Bedrock AgentCore 评估接入决策模型

算力 来源:AWS Machine Learning 2026-11-05 02:46 0 阅读 机器翻译
Amazon Bedrock AgentCore 评估接入决策模型

在评估智能体时,你提出的一些问题是狭窄且可重复的。这个回复是否遵守了策略?这个工具调用批准起来是否安全?这个答案是否与检索到的上下文相符?决策模型回答的正是这类问题。它们接收结构化状态和一组固定的允许答案,并返回在这些答案上的校准概率,而不是输出那种每次运行置信度都会漂移的自由文本。这让它们非常适合充当智能体评估中的打分环节——在这个环节,你想要的正是可以设定阈值、值得信赖的标签和数值。

Amazon Bedrock AgentCore Evaluations 是 Amazon Bedrock AgentCore 的一项能力,可在智能体的整个生命周期中对其打分:开发期间按需评估、针对已存储会话的批处理任务评估,以及针对生产流量的在线评估。除了 15 个内置评估器之外,AgentCore Evaluations 还支持自定义评估器,你可以将其定义为 LLM-as-a-judge 或基于代码的评估器。自定义 LLM-as-a-judge 评估器会使用 Amazon Bedrock 中的某个模型来应用你的指令和评分标准。基于代码的评估器则在 AWS Lambda 函数中运行你自己的逻辑,该逻辑可以调用你选择的外部系统、模型或 API,包括决策模型。

决策模型适用于答案固定、了解情况的人一眼就能判断的问题,例如智能体是否拒绝了某个请求,或者回复是否切题。在你用自己的流量验证过某个问题之后,决策模型对于该项判断或打分可能已经足够,你也可以把同样的方法扩展到你的策略中特定的有害性或刻板印象类别。由于这两种后端都能在一次请求中回答关于同一状态的多个问题,你可以把它们当作筛选层而非替代方案,在你的各项指标上给出通过/不通过信号,从而挑出值得由内置评估器做更深入评估的轮次——内置评估器给出的解释能帮助你排查问题出在哪里。

本文介绍如何通过基于代码的评估器,把两种决策模型后端接入 AgentCore Evaluations。第一种是 Jev,来自 TypeSafe 的托管决策模型。第二种是 Strands Decider 2B,一个由你自己运行的开源决策模型。读完本文,你将了解如何把 AgentCore 评估器的输入转换为决策模型请求,以及如何把模型的概率转回 AgentCore 的评估结果。不过首先要问的是:什么时候该用自定义的 LLM-as-a-judge 评估器,什么时候决策模型更合适?下一节将拆解什么是决策模型,以及其中的取舍。

决策模型是一种小型专用模型,经训练后从一组固定选项中做出选择,并返回经过校准的置信度分数,而不是生成自由形式的文本。通用大语言模型(LLM)评估器会给出一个分数并附上理由,而决策模型则输出在预定义答案上的概率分布。这让你能获得直接的置信度度量,可以设定阈值、进行聚合,并以程序化方式据此采取行动。因此,决策模型非常适合那些判断可以被表述为固定选项之一的选择的评估任务,例如“该回答是否有依据?”或“智能体是否完成了用户的目标?”

当你希望在标签之外还获得一个校准良好的置信度分数,而不仅仅是二元判断时,决策模型最为有用。当你希望始终一致地执行某一套特定评分标准,而不依赖通用 LLM 去解读它时,决策模型同样适用。

TypeSafe 的托管决策模型 Jev,以及 Strands Agents 团队推出的开源 20 亿参数决策模型 Strands Decider,都支持三类问题。noul 问题提出一个是/否问题,并返回一个介于 0(否)与 1(是)之间的概率。选择问题询问最多 255 个选项中哪一个适用,返回最可能的选项以及完整的概率分布和置信度。评分问题询问 2 到 10 个有序等级中哪一个适用,返回概率加权分数、各等级上的概率分布以及置信度。许多评估评分标准都符合其中某一类。“该回答中的每一个事实是否都得到工具结果的支持?”是一个是/否问题,而“按四级评分标准,该回答有多大帮助?”则是一个评分问题。

AgentCore 中的内置评估器目前使用通用 LLM,依据评分标准评估智能体响应,并解释其判断。当评估需要串联证据、识别缺失信息,或在解释与分数同样有用的任务中深入分析响应时,这很有价值。决策模型并不是 LLM 评估器的替代品。它们是由基于代码的评估器框架所实现的一种补充。可将它们用于范围狭窄、定义明确的检查,与内置评估器范围更广的评估配合使用。

为了展示这些评估器类型如何协同工作,本文构建了一个端到端示例:一个 HR Assistant 智能体,由确定性的基于代码的检查、内置的 LLM-as-a-judge 评估器,以及由 Jev 或 Strands Decider 2B 支持的决策模型评估器进行评估。

该智能体使用 Strands Agents 构建,并部署到 AgentCore runtime,这是 Amazon Bedrock AgentCore 的一项能力。它通过五个工具回答员工问题,这些工具返回固定的模拟数据,例如带薪休假(PTO)余额、PTO 请求、HR 政策、福利和工资单。在此代码示例中,我们使用固定数据,以便直接验证 LLM-as-a-judge 和决策模型评估器的行为。当你额外拥有真实基准(ground truth)时,可以确认每个评估器是否捕捉到正确的失败。此代码示例可帮助你搭建框架,以便针对自己的用例测试决策模型与 LLM-as-a-judge 的对比。

图 1:HR Assistant 的端到端架构,由基于代码的检查、内置评估器,以及由 Jev 或 Strands Decider 支持的决策模型评估器进行评估

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Use decision models with Amazon Bedrock AgentCore Evaluations)

Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a structured state and a fixed set of allowed answers and return a calibrated probability over them, rather than free-form text whose confidence drifts from run to run. That makes them a strong fit for the scoring step in an agent evaluation, where you want a label and a number you can threshold and trust.

Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, AgentCore Evaluations supports custom evaluators that you define as either an LLM-as-a-judge or a code-based evaluator. Custom LLM-as-a-judge evaluators apply your instructions and rating scale with a model in Amazon Bedrock. Code-based evaluators run your own logic in an AWS Lambda function, and that logic can call your choice of external system, model, or API, including a decision model.

Decision models suit questions with a fixed set of answers that an informed person could decide at a glance, such as whether the agent refused a request or whether a response stayed on topic. After you validate a question on your own traffic, a decision model can be sufficient for that decision or score, and you can extend the same approach to specific harmfulness or stereotyping categories in your policy. Because both backends can answer several questions about the same state in a single request, you can use them as a screening layer rather than a replacement, running a pass or fail signal across your metrics to select the turns that warrant deeper assessment by the built-in evaluators, whose explanations help you investigate what went wrong.

This post shows how to connect two decision model backends to AgentCore Evaluations through code-based evaluators. The first is Jev, a hosted decision model from TypeSafe. The second is Strands Decider 2B, an open-source decision model you run yourself. By the end of this post, you will know how to convert the AgentCore evaluator input into a decision model request, and the model’s probabilities back into an AgentCore evaluation result. First, though, when should you reach for a custom LLM-as-a-judge evaluator, and when does a decision model fit better? The next section breaks down what a decision model is and the trade-offs involved.

Decision models

A decision model is a small, specialized model trained to select from a fixed set of options and return calibrated confidence scores rather than generating free-form text. A general-purpose large language model (LLM) evaluator produces a score with a rationale. A decision model instead produces a probability distribution over predefined answers. That gives you a direct confidence measure you can threshold, aggregate, and act on programmatically. This makes decision models well-suited for evaluation tasks where the judgment can be stated as a choice among a fixed set of options: “Is this response grounded?” or “Did the agent complete the user’s goal?”

Decision models are most useful when you want a well-calibrated confidence score alongside the label, not just a binary judgment. They are also appropriate when you want to enforce a specific rubric consistently without relying on a general-purpose LLM to interpret it.

Jev, the hosted decision model from TypeSafe, and Strands Decider, an open-source, 2-billion-parameter decision model from the Strands Agents team, both support three types of questions. A noul question poses a yes/no question and returns a probability between 0 (no) and 1 (yes). A choice question asks which of up to 255 options applies, returning the most probable option along with the full distribution and a confidence. A score question asks which of 2 to 10 ordered levels applies, returning a probability-weighted score, the distribution over levels, and a confidence. Many evaluation rubrics fit one of these types. “Is every fact in this response supported by the tool results?” is a yes/no question, and “How helpful was this response, on a four-level rubric?” is a score.

Decision models versus LLM-as-a-judge

The built-in evaluators in AgentCore today use a general-purpose LLM to assess the agent response against a rubric and explain its judgment. This is valuable when the evaluation requires connecting evidence, identifying missing information, or working through a response in depth in tasks where the explanation is as useful as the score. Decision models are not a replacement for LLM evaluators. They’re an addition that the code-based evaluator framework makes possible. Use them for narrow, well-defined checks alongside the built-in evaluators’ broader assessment.

Solution overview

To show how these evaluator types work together, this post builds an end-to-end example of an HR Assistant agent evaluated by deterministic code-based checks, built-in LLM-as-a-judge evaluators, and decision-model evaluators backed by Jev or Strands Decider 2B.

The agent is built with Strands Agents and deployed to AgentCore runtime, a capability of Amazon Bedrock AgentCore. It answers employee questions using five tools that return fixed mock data such as paid time off (PTO) balances, PTO requests, HR policies, benefits, and pay stubs. We use fixed data in this code sample to make it straightforward to verify evaluator behavior from LLM-as-a-judge and decision models. When you additionally have ground truth, you can confirm whether each evaluator catches the right failures. This code sample helps you set up the framework for testing decision models compared to LLM-as-a-judge for your own use cases.

Architecture of the HR Assistant agent on AgentCore runtime, evaluated by code-based, built-in, and decision-model evaluators

Figure 1: End-to-end architecture of the HR Assistant evaluated by code-based checks, built-in evaluators, and decision-model evaluators backed by Jev or Strands Decider

Prerequisites

To deploy the sample, you need the following:

  • An AWS account in a Region where AgentCore Evaluations is available. The sample defaults to US West (Oregon).
  • Amazon CloudWatch Transaction Search enabled in the account.
  • Access to a foundation model in Amazon Bedrock.
  • Python 3.12 or later, the AWS Command Line Interface (AWS CLI), and Docker.
  • Jev requires a TypeSafe API key. Visit typesafe.ai for access details.
  • Strands Decider requires a host running the Strands Decider server. You can run it locally for testing, or on Amazon Elastic Compute Cloud (Amazon EC2) inside your virtual private cloud (VPC) so the Lambda function can reach it.

Seeing it in action

You can find the full code sample in the AgentCore samples GitHub repository. Two Python scripts handle everything:

  • ../utils/deploy.py deploys the HR Assistant to AgentCore runtime.
  • evaluate.py deploys evaluators, invokes the agent, and runs the evaluation modes.

Running evaluate.py without flags executes five steps:

  1. Creates the Lambda execution role (AgentCoreLambdaEvaluatorRole).
  2. Packages and deploys two deterministic Lambda evaluators: hr-response-length (a trace-level evaluator that checks whether the agent’s response stays within length bounds) and hr-fact-checker (a session-level evaluator that regex-validates HR facts against the known mock data).
  3. Registers both as AgentCore code-based evaluators through the control plane.
  4. Runs on-demand evaluation with EvaluationClient and OnDemandEvaluationDatasetRunner across five labeled scenarios, combining the code-based evaluators with Builtin.Correctness, Builtin.Helpfulness, and Builtin.ResponseRelevance.
  5. Creates an online evaluation configuration that continuously scores live sessions at 100 percent sampling.

At this point you have a working evaluation pipeline with deterministic checks and LLM-as-a-judge scoring. Decision-model evaluators are opt-in. You can add them with a flag, and they run alongside the base evaluators with no changes to the existing flow. Using the flags --with-jev --jev-secret-arn <ARN> deploys jev-evaluator and registers three evaluators: JevGroundedness, JevHelpfulness, and JevGoalCompletion. Using the flag --with-decider deploys strands-decider-evaluator and registers the same three evaluators backed by Strands Decider 2B. Add --decider-ec2 to auto-provision the Amazon EC2 instance, security groups, IAM role, and Lambda VPC configuration without manual console steps.

The evaluation pipeline that evaluate.py deploys, combining deterministic, built-in, and decision-model evaluators across on-demand and online modes

Figure 2: The evaluation pipeline deployed by evaluate.py, from deterministic checks through built-in and decision-model evaluators

Each decision-model backend contributes three evaluators:

Evaluator Level Question type Label Value
JevGroundedness / DeciderGroundedness TRACE noul Grounded or Ungrounded P(grounded)
JevHelpfulness / DeciderHelpfulness TRACE score (4-level) Most probable level Expected level, scaled 0–1
JevGoalCompletion / DeciderGoalCompletion SESSION choice (3-way) Most probable option Probability-weighted value

Build the evaluator with decision models

Using the flags --with-jev and --with-decider creates a custom code-based evaluator, where you have complete control of the code you want to run as your evaluator. We create AWS Lambda functions where the decision model is invoked inside an AWS Lambda function and returns a label or a numeric value. This Lambda function is registered as the code-based evaluator in AgentCore Evaluations.

The Lambda that is registered as the evaluator receives OpenTelemetry spans of the agent. The spans.py module normalizes the shapes OpenTelemetry content can arrive in and then formats it appropriately to make the call to the decision model. For trace-level evaluation, only the targeted turn and brief prior history are sent, whereas the full conversation is included for session-level.

The code-based evaluator Lambda converting OpenTelemetry spans into a decision-model request and mapping the probabilities back to an AgentCore result

Figure 3: How the code-based evaluator Lambda adapts OpenTelemetry spans for the decision model and maps the result back to AgentCore

The adapter converts the model’s probability distribution into a result that AgentCore evaluation expects. For a noul question, the probability of true becomes the value and the most probable answer becomes the label. For a score question, the value is the probability-weighted expected level scaled to [0, 1]. For a choice question, it is the weighted average of predefined option values.

Validate a decision-model evaluator on your own traffic

When you rewrite a rubric as a decision model question, you’ve created a new measurement rather than a copy of the old one. A reframed rubric often lands stricter or looser than the original intended, and you won’t see that drift until you check it against real sessions. Before you gate a release on a decision-model evaluator, validate it the same way you’d validate any new measurement, against a labeled sample of your own traffic. The goal of this section is to walk you through that validation end to end, so you finish with an evaluator you can threshold with confidence.

The procedure has five steps.

Step 1. Build a labeled sample. Pull a set of stored sessions from your agent’s traffic, with enough coverage of the failures and edge cases you care about. A sample that covers only the happy path tells you nothing about where the evaluator breaks.

Step 2. Get human reference labels. Have three independent labelers score each turn for the property your question measures and take the majority vote as the reference label. Keep track of which turns were unanimous and which were split. Although unanimous turns are the clear-cut cases, the split-vote turns are the valuable ones, because they reveal ambiguity in the rubric itself before the model ever sees it.

Step 3. Run both evaluators on the same inputs. Run the decision-model evaluator and the matching built-in evaluator over those same sessions in a single batch evaluation. Scoring identical inputs is what makes the comparison apples to apples.

Step 4. Read the results through three lenses. With the majority-vote labels and the easy and hard split in hand, you can now answer the three questions that tell you whether the evaluator is ready.

  • Agreement rate: does it match your humans? Measure how often each evaluator agrees with the majority-vote labels, and do it per question, since one question’s accuracy says nothing about another’s. Then break the numbers out by easy (unanimous) and hard (split-vote) cases. An evaluator that stumbles only on the hard cases is behaving a lot like your human labelers did, which is exactly what you want.
  • Calibration: do the confidence scores mean something? Group turns by the evaluator’s reported probability and compare each group’s agreement with the labels. Higher confidence should track higher accuracy. Then check whether easy cases consistently draw higher confidence than hard ones. If they do, the model’s uncertainty is tracking genuine ambiguity, and you can use those groups to pick a decision threshold per evaluator instead of defaulting to 0.5.
  • Disagreement analysis: where and why do they differ? Find the turns where the two evaluators disagree with each other, and check whether those are the same turns your humans found ambiguous. Read them. Most disagreements trace back to the wording of the question, and a revised question clears them up. When the errors cluster in the hard cases, the rubric is the problem rather than the model.

Step 5. Re-run the check when anything changes. Treat this as a repeatable check rather than a one-time gate. Any time you reword a question or move to a new model version, run it again. Each result’s explanation records the model version that produced it, so you know which version a given score came from.

Choosing the best set of evaluators for your agent

The built-in evaluators remain the foundation of broad evaluation strategy in AgentCore Evaluations. They assess the response against a rubric and its supporting context, and explain the judgment so developers can investigate what went wrong. For the HR Assistant, a faithfulness evaluation can identify a claim that contradicts a tool result, whereas a helpfulness evaluation can point out that the agent answered a balance question but left a requested action incomplete.

Faithfulness, helpfulness, and correctness can require connecting evidence, identifying missing information, or working through a solution. The preprint JEV-as-a-Judge: Accept When Confident, Escalate When Unsure describes a related distinction as “reading versus deriving.” A narrow decision-model check can flag a turn for attention, whereas the built-in evaluator provides the broader assessment and reasoning needed to review it. Decision models shine brightest when you treat them as a gate rather than a replacement. The cascade pattern works like this. First, run a fast, calibrated decision model on every turn of every session to get a cheap pass/fail signal. Then invoke a full LLM evaluator only on sessions where the signal falls below your threshold.

The cascade pattern, where a decision model screens every turn and escalates low-confidence turns to a full LLM evaluator

Figure 4: The cascade pattern, where a calibrated decision model screens every turn and only low-confidence turns escalate to a built-in evaluator

Criterion LLM as Judge Decision Models as Judge
Output Score + explanation Calibrated probability
Latency 2–10 s ~hundreds of ms (network)
Customizable Limited (fixed prompt templates) Question definitions
Best for Qualitative nuance, explanation Calibrated judgment

We recommend the following setup:

  1. Configure online evaluation with the built-in evaluators you rely on, as the standard measure of quality across your agents.
  2. Add decision-model evaluators for domain-specific yes/no and choice questions to the same configuration, along with deterministic code-based evaluators for anything you can compute exactly.
  3. Each week, or on whatever schedule suits your release cycle, query the online results for sessions where a decision-model check failed or scored near its threshold, and run batch evaluations on those sessions with the built-in evaluators to get explanations.
  4. Gate releases on the built-in evaluators and on the decision-model evaluators you have validated against labeled data, and track how often the two agree.

Clean up

To avoid ongoing charges, run the cleanup.py that deletes the agent hosted on AgentCore runtime, the evaluators and evaluation config, the AWS Lambda functions that are registered as the evaluators, the EC2 instance that hosts the Strands Decider model, AWS Secrets Manager key and the associated log group on CloudWatch.

Conclusion

In this post, you connected two decision models, Jev and Strands Decider, to AgentCore Evaluations using code-based evaluators, and the accompanying code sample gives you a framework for testing those models and seeing where they add value as evaluators. The built-in evaluators remain your baseline. Decision models add narrow, validated checks that come with a measure of certainty you can threshold and act on. The practical path forward is to add decision-model checks alongside the built-in evaluators you already run, measure how often they agree with your reference labels on your own traffic, and expand coverage only once you trust them.

To get started, review the Amazon Bedrock AgentCore Evaluations documentation, integrate custom evaluators using code-based evaluators and take a deeper look into Strands Decider model. Get started with code samples by deploying the sample evaluators from the Amazon Bedrock AgentCore sample repository.

 


About the authors

Sungyeon Kim

Sungyeon Kim

Sungyeon is an Applied Scientist at AWS, where he conducts research and develops machine learning solutions for agentic applications. His research interests include language model training and evaluation, generative AI, and agentic systems. He has worked across various AI domains, including computer vision, multimodal learning, and retrieval systems, in both academia and industry.

Bharathi Srinivasan

Bharathi Srinivasan

Bharathi is a Generative AI Data Scientist at AWS. She is passionate about Responsible AI to increase the reliability of AI agents in real world scenarios. Bharathi guides internal teams and AWS customers on their responsible AI journey. She has presented her work at various machine learning conferences.

Jonathan Buck

Jonathan Buck

Jonathan is a Principal Engineer at AWS. His work focuses on infrastructure and tooling to support AI agent environments and evaluation. Jonathan works with both research and engineering teams to shape the direction of next-generation AI products and services for AWS customers.

Haibo Ding

Haibo Ding

Haibo is a Principal Applied Scientist and Science Manager working on agentic AI at Amazon. He holds a Ph.D. from the University of Utah. His work focuses on large language models (LLMs) and AI agents, where he leads research in areas such as LLM post-training, benchmarking, agent evaluation, and tool optimization. He has served as an area chair for conferences such as AAAI and ACL, and previously as Program Chair for KDD 2025 Workshop on Prompt Optimization.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 AWS Machine Learning(Use decision models with Amazon Bedrock AgentCore Evaluations),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗