用 Pensive AI 让故障诊断变得可靠

作者:Praneeth Yenugutala、Victor Dusa、Anastasia Samoilova、Megan Ren、Justin Wei、Marcelo Mayworm
在 Netflix,数据与机器学习从业者每天通过 Titus、Spark、Trino 和 Druid 等编排系统与计算引擎运行数百万个工作负载。在高峰日,大约有 100,000 个工作负载失败。当工作负载失败时,失败本身只是开始。仍然需要有人从多个日志中找到有用的信号,判断原因是出在自己的代码还是平台,并决定是重试、重新配置还是上报。即便只有一小部分难以处理的失败,也可能消耗工程时间、浪费算力,并中断关键数据管道。
我们对该问题的第一个答案是 Pensive,这是 Netflix 数据平台中的自动诊断与修复系统。Pensive 从失败的工作负载中收集日志,使用正则表达式和手写 Java 规则来提取并分类错误,然后将重试或修复指导传递给 Maestro 等产品。它使约 50% 的失败实现了自助处理,在某些情况下还能自动处理。
规则对于已知模式效果很好,但只有当有人识别出新的失败模式、理解日志、编写规则,并在引擎和日志格式变化时保持规则更新,其覆盖范围才会扩大。在一个为期三个月的周期内,Pensive 处理了大约七百万次作业失败分类,但其中约一半落入了未知类别。我们需要一种方法来扩大覆盖范围,同时不放弃 Pensive 中已经奏效的确定性部分。
为了解决 Netflix 规模下这一模糊而复杂的问题,我们投入建设了 Pensive AI,为数据和 ML 从业者提供 AI 辅助的错误提取,以及有针对性的诊断与修复指导。目标并不是简单地把一个模型摆在大量日志前面。我们需要有边界的提取、可预测的评估输入、响应及时的产品行为,以及能够贯穿从本地开发到生产全流程的质量检查。这些要求塑造了设计的每一层,我们将在本文中逐一讨论。Pensive AI 将 AI 包裹在确定性流程周围,从而在整个软件开发生命周期中带来更广的覆盖范围、低延迟、可控的 AI 成本以及更易维护的特性。在这一基础之上,我们计划探索更多智能体式调试体验,第 6 部分将对此进行介绍。
早期结果令人鼓舞。AI 错误提取现在覆盖 100% 的失败,包括旧版 Pensive 归类为 unknown 的失败。在无人使用 Pensive AI 的失败步骤尝试中,手动探索调试日志的操作次数平均比有人使用的情况多 321%(这是一项观察性对比,并非因果结论)。
这两条路径相互补充。错误提取改进了已有的能力,并且始终开启。它让 Maestro 工作负载失败的用户能快速查看日志中可能指向直接根因的部分。它延迟低、成本低,在很大一部分场景中都能帮上忙。按需调试则带来了一项新能力。它会审视日志中的各种症状,并加入 LLM 自身的知识与推理进行更深入的分析,延迟和成本略高。提取为每一次失败提供一次快速的初步查看,而当这次初步查看不够时,由用户触发的按需调试会进一步深入。
传统的 Pensive 规则很精确,但每一条规则只捕捉一种特定的日志形态。简化后的规则大致如下:
- name: SparkContainerOOM logSource: GENIE_STDERR restartable: false regex: "Container.*is running beyond physical memory limits.*Killing container" summary: Spark AM container runs out of memory借助 Pensive AI,我们希望找出那些没有对应规则的有用错误,并在平台升级后熟悉的错误形态发生变化时更容易地适配。Data Portal(我们的数据管道 UI)会在可用时呈现这些提取出的错误,并链接回源日志,为从业者提供一个聚焦的分诊起点。这就引出了一个问题:我们怎样才能让模型识别出错误,同时又不必让它逐个 Token 地重复一长串堆栈跟踪?
生成输出 Token 的速度远慢于读取输入 Token。若采用直接转写的方式,一旦出现长错误,代价高昂且速度迟缓。因此,Pensive 会为每条日志的行编号,并向模型提供一系列 {lineNumber, text} 记录。模型识别出错误区间的首行与末行,Pensive 再根据自己保存的日志副本重建所选文本,从而降低成本、改善延迟。这样一来,无论选中的日志跨越几行还是数百行,模型始终只返回两个整数(起始行号和结束行号)。展示的内容直接来自原始日志,减少了幻觉。模型仍可能选错范围,这正是提取质量始终是我们 AI 评估中一项核心指标的原因,我们稍后会讨论这一点。
对于编排好的工作流,只要某个步骤失败,“Pensive AI Error Extraction”就会自动运行。一次步骤尝试(step attempt)是指 Maestro 工作流中某个步骤的一次执行。Maestro 本就会调用 Pensive 进行基于正则表达式推导的错误分类与可重启性分类,因此 AI 提取会在同一请求中运行,并与既有字段一并返回结果。对于失败的步骤尝试,Pensive 的整体响应必须在 Maestro 硬性的 90 秒后处理超时内返回。
这一时限决定了提取路径的设计。Pensive 会并行地从各条日志中提取错误以降低延迟,各条日志的调用共享同一个截止时间,且该时间设定得远早于 Maestro 的超时上限。某条日志的提取调用若未赶上截止时间,就会被排除在结果之外,从而使整体请求能够按时返回。
Pensive AI 与各类消费方之间的数据契约也需要同样审慎对待。旧有的数据模型只包含一个扁平化的错误字符串;而 AI 抽取则返回一个错误列表,每个日志来源对应一个错误,并带有指回原始日志的深层链接。就地修改旧字段会破坏现有客户端。我们在旧的非 AI 错误抽取数据模型旁边引入了新的数据模型,并继续同时产出两者。Data Portal 是我们用于监控和观测数据管道的用户界面,在灰度发布期间,它会优先采用 AI 错误,在不可用时回退到旧值。这实现了渐进式发布,也确保历史故障仍能显示错误,因为我们出于日志留存方面的考虑,选择不回填旧数据。
迁移至今仍在进行中。用于展示失败 Maestro 步骤的 Data Portal UI 界面现在读取新的 AI 抽取错误字段,而查询编辑器(用于编辑和运行 SQL)、Workbench notebook(托管计算平台及用于编辑和运行基于 jupyter 的 notebook 的 UI)以及部分调用 Pensive 的 Slack 客户端仍使用旧的错误抽取路径。每个客户端都需要先完成兼容的 schema 变更,才能消费新数据。让两个字段都保持有值,使各团队可以按自己的节奏采用新行为,而不必在我们持续演进 Pensive AI 的过程中被迫进行协调一致的统一切换。
除错误抽取之外,我们还构建了 Pensive AI Debugging,这一工具可分析失败的步骤尝试,并提供根因分析和建议的修复方案。它可通过编程 API、MCP 以及产品 UI(Data Portal)使用,全部由 Netflix 数据平台上统一的入口点支撑。这一共享模型支持一致的行为和定制化的产品体验。
这种诊断仍须快速生成,并在各次排查之间保持一致。生成一条调试响应可能需要数十秒甚至一分钟,而错误提取只需几秒。我们按需运行完整的基于 AI 的调试,而非自动运行,从而将成本控制在合理范围内,因为大多数失败的步骤从不会被打开。我们以失败的步骤尝试,而非触发它的人,作为工作单元,因此任何排查同一次失败尝试的人看到的都是相同的诊断。
这一选择为我们提供了稳定的缓存工作单元。我们按步骤尝试缓存每一条调试响应。第二次请求就变成一次键值读取,而不再是又一次 AI 推理调用。缓存还能避免重复工作,进一步降低延迟和成本。这也意味着在 Data Portal 中生成的调试结果会被缓存,MCP 路径可直接立即取用,而无需再产生一次 AI 推理调用。条件写入会创建一条“进行中”记录,由一个调用方启动生成,其他调用方等待共享结果,从而在同一步骤尝试失败同时收到并发请求时避免冲突。如果生成过程超过弃置超时仍处于进行中状态,Pensive 会将该记录视为已弃置并回收。已完成的响应会在 90 天后过期,以帮助控制成本并优化存储,因为针对久远得多的工作负载的调试请求并不常见。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Making Failure Diagnosis Reliable with Pensive AI)
By Praneeth Yenugutala, Victor Dusa, Anastasia Samoilova, Megan Ren, Justin Wei, Marcelo Mayworm
At Netflix, data and machine learning practitioners run millions of workloads each day across orchestration systems and compute engines such as Titus, Spark, Trino, and Druid. On peak days, roughly 100,000 workloads fail. When workloads fail, the failure itself is only the beginning. Someone still needs to find the useful signal in several logs, decide whether the cause sits in their code or the platform, and determine whether to retry, reconfigure, or escalate. Even a small share of difficult failures can consume engineering time, waste compute, and interrupt critical data pipelines.
Our first answer to that problem was Pensive, an auto-diagnosis and remediation system in Netflix’s data platform. Pensive collected logs from failed workloads, used regular expressions and hand-authored Java rules to extract and classify errors, then passed retry or remediation guidance to products such as Maestro. It made ~ 50% of failures self-service and, in some cases, automatic.
Rules work well for known patterns, but their coverage grows only when someone recognizes a new failure mode, understands the logs, writes a rule, and keeps it current as engines and log formats change. Over one three-month period, Pensive processed roughly seven million job-failure classifications, yet about half fell into the unknown bucket. We needed a way to expand coverage without giving up the deterministic parts of Pensive that already worked.
To solve this ambiguous and complex problem at Netflix scale, we invested in Pensive AI to add AI-assisted error extractions and focused diagnosis and remediation guidance to our data and ML practitioners. The goal was not simply to place a model in front of a large collection of logs. We needed bounded extraction, predictable evaluation inputs, responsive product behavior, and quality checks that could follow the system from local development through production. Those requirements shaped every layer of the design, which we will discuss in this blog. Pensive AI wraps AI around deterministic processes, which gives broader coverage, low latency, controlled AI costs, and easier maintenance across the software development lifecycle. Building on this foundation, we plan to explore more agentic debugging experiences, as Part 6 describes.
Early results are encouraging. AI error extraction now covers 100% of failures, including the failures that legacy Pensive classified as unknown. Failed step attempts where no one used Pensive AI averaged 321% more manual debug-log exploration actions than attempts where someone did (an observational comparison, not a causal claim).

The two paths complement each other. Error extraction improves an existing capability and is always on. It lets the user of a failed Maestro workload quickly check the parts of the logs that may point to an immediate root cause. It has low latency and low cost, and it can help in a large share of cases. On-demand debugging enables a new capability. It looks at symptoms across the logs and adds the LLM’s own knowledge and reasoning for a deeper analysis, at slightly higher latency and cost. Extraction gives a fast first look at every failure, and on-demand debugging triggered by the user goes deeper when that first look is not enough.
Part 1: Finding the useful errors
Legacy Pensive rules are precise, but each one captures a particular log shape. A simplified rule looks like this:
- name: SparkContainerOOM
logSource: GENIE_STDERR
restartable: false
regex: "Container.*is running beyond physical memory limits.*Killing container"
summary: Spark AM container runs out of memory
With Pensive AI, we wanted to find useful errors that had no corresponding rule and adapt more easily when familiar errors changed shape after a platform upgrade. Data Portal (our UI for data pipelines) surfaces these extracted errors when available and links back to the source logs, giving practitioners a focused starting point for triage. That raises a question: how can we ask a model to identify an error without asking it to repeat a long stack trace token by token?
Producing output tokens is much slower than reading input tokens. A direct transcription approach would make long errors expensive and slow. Instead, Pensive numbers the lines of each log and gives the model a sequence of {lineNumber, text} records. The model identifies the first and last lines of the error span, and Pensive reconstructs the selected text from its own log copy, leading to lower cost and better latency. This way, the model always returns two integers (start and end line number), whether the selected logs span a few lines or hundreds of lines. The displayed content comes directly from the original log, reducing hallucinations. The model can still choose the wrong range, which is why extraction quality remains an integral measurement for our AI evals, as we will discuss later.
For orchestrated workflows, “Pensive AI Error Extraction” runs automatically whenever a step fails. A step attempt is one execution of a step within a Maestro workflow. Maestro already calls Pensive for a regex-derived error and restartability classification, so AI extraction runs in the same request and returns its results alongside the established fields. The overall Pensive response must return within Maestro’s hard 90-second post-processing timeout for a failed step attempt.

That deadline shaped the extraction path. Pensive extracts errors from the logs in parallel to reduce latency, and the per-log calls share one deadline set well inside Maestro’s timeout limit. A per-log extraction call that misses the deadline is excluded from the result, allowing the overall request to return on time.
The data contract between Pensive AI and the various consumers needed the same care. The legacy data model contained one flattened error string; AI extraction instead returns a list of errors, one for each log source, with deep links back to the originating logs. Changing the old field in place would have broken existing clients. We introduced the new data model beside the legacy non-AI error extraction data model and continued producing both. Data Portal, our user interface for monitoring and observing data pipelines, preferred AI errors when available and fell back to the legacy value when they were not during the rollout. That allowed a gradual rollout and ensured that historical failures still display an error, since we chose not to backfill the old data due to log retention issues.
The migration is still in progress today. The Data Portal UI surface used for failed Maestro steps reads the new AI extracted error field today, while the query editor (for editing and running SQL), Workbench notebooks (managed compute platform and UI to edit and run jupyter based notebooks) and some slack clients calling Pensive still use the legacy error extraction path. Each client needs a compatible schema change before it can consume the new data. Keeping both fields populated lets teams adopt the new behavior at their own pace without forcing a coordinated cutover as we continue evolving Pensive AI.
Part 2: Full failure diagnosis and the product experience
Beyond error extraction, we built Pensive AI Debugging, a tool that analyzes failed step attempts and provides root-cause analysis and suggested remediations. It is available through programmatic APIs, MCP, and the product UI (Data Portal), all backed by a common entry point across the Netflix Data Platform. This shared model supports consistent behavior and custom product experiences.
This diagnosis still has to arrive quickly and remain consistent across investigations. A debugging response can take tens of seconds or a minute to generate, compared with a few seconds for error extraction. We run full AI-based debugging on demand instead of automatically, which keeps costs under control because most failed steps are never opened. We treat the failed step attempt, not the person who triggered it, as the unit of work, so everyone investigating the same failed attempt sees the same diagnosis.
That choice gives us a stable unit of work for caching. We cache each debugging response by step attempt. A second request becomes a key-value read instead of another AI inference call. The cache also prevents duplicate work, reducing latency and lowering costs further. This also means that a debugging generated in the Data Portal is cached and immediately retrieved for the MCP path instead of incurring another AI inference call. A conditional write creates an in-progress record, one caller starts generation, and other callers wait for the shared result to avoid conflicts when concurrent requests come in for the same step attempt failure. If generation remains in progress past an abandonment timeout, Pensive treats the record as abandoned and reclaims it. Completed responses expire after 90 days to help control costs and optimize storage because debugging requests for much older workloads are uncommon.
Pensive AI supports multiple ways to initiate debugging responses. A caller can (1) read a cached response, (2) generate one if no cache entry exists, or (3) regenerate and overwrite an existing response. Those semantics support the product UI, programmatic access, and the evaluation harness while keeping the response associated with a single immutable workload failure.

For users, the most important behavior is that a long-running answer feels alive. Pensive streams the response as it is produced. The service emits server-sent events, and the GraphQL layer republishes them as a WebSocket subscription for the Data Portal client.
Each event carries the complete response generated so far. That duplicates a small amount of text, but it keeps the client simple: it does not need a replay buffer or token reassembly logic, and the next event corrects the display after a dropped or late event. Generation is independent of the connection that requested it. A disconnect does not cancel the work, and the final answer still reaches the cache. A hard reload can show a loading state and poll for the completed result. Within the single-page application backing the Data Portal UI, active streams are maintained independently of view and component lifecycle, so navigation does not interrupt a response in progress.
Part 3: Measuring and improving the behavior we want
LLM-assisted systems need more than a demo or a few working examples. Before shipping in production, we need confidence in both error extraction and debugging guidance, and we need to know that we have not regressed on failures that legacy Pensive already handled. We built separate evaluation frameworks focused on the two distinct jobs.
Building labeled data is often the expensive part of evaluation. For error extraction, legacy Pensive’s hand-written rules provide reference labels for failures they recognize. They also serve as excellent ground truth in most cases. We run those rules over production logs and compare their extracted spans with Pensive AI’s output. This gives us a reliable regression set without relabeling every example by hand, though it is not an absolute ground truth (since our human-built rules could also have been stale). For now, we keep these rules frozen as a label source. This lets us continue to build comparison evals and run them on top of the existing system.
Stable inputs are essential for repeatable evaluation experiments where we A/B test new models/configurations or context injections to improve response quality. The logs Pensive AI uses can expire after as few as seven days or as many as 30 days. Production Pensive AI requests archive fetched logs to S3 object storage asynchronously during workload execution time so the upload is not on the user-facing critical path. Uploads do not overwrite an existing object, preserving the exact snapshot used to create an evaluation case. Evals use those archived logs, so score changes remain attributable to the system under test. The archive also prevents parallel experiments from overloading the source systems and lets us run them without affecting production. Cached S3 log inputs further reduce costs when running evaluations.
Concretely, we measure four complementary signals for error extraction:
- Inclusion: An LLM judge compares the legacy Pensive error for a failure with the errors Pensive AI extracted and determines whether the known error is still represented, even when wording differs.
- Precision: A second judge sees only the extracted AI errors and evaluates whether each is meaningful and actionable on its own.
- Repeated noise: A script looks across the whole dataset for lines that recur too often. A generic status line may look harmless in one answer but become distracting if it appears in a large share of failures. Lines appearing in more than about 30% of workflows are candidates for explicit exclusion. In practice, the exclusion list has only three lines, each of which otherwise appeared in nearly every extracted error.
- Latency: We record the latency of every extraction call. Quality scores alone cannot reveal when an apparent improvement requires too much additional work, which often correlates with increased AI costs.
This combination gives us an actionable feedback loop. Prompt changes can reduce general noise, while the dataset-level view exposes the stubborn boilerplate that prompt changes do not eliminate. Existing rules act as a regression floor, and archived logs give us a repeatable way to compare changes. At Pensive AI’s scale, model and system choices must satisfy quality, latency, maintainability and cost constraints together.
To seed the debugging ground-truth dataset for evaluation, we mine internal engineering-support Slack threads and use an LLM to identify threads that link to an orchestrated workflow, describe a resolved problem, and match the class of issue Pensive AI handles. A second model synthesizes a reference debugging response from the relevant discussion. Paired with logs from that workflow failure, it becomes a concrete evaluation row in our evals dataset. This approach works only retrospectively, because Pensive AI in production aims to provide debugging guidance before such a Slack discussion happens.
We also made the debugging generation process visible. Tracing follows the request through Pensive, with child spans for each log fetch and extraction tool call. Our tracing backend based on Braintrust brings the prompt, completion, tool calls, and latency into a single view. During early product development, that visibility helped us inspect low-performing debugging responses in context instead of guessing which input or operation caused them. Tracing helped us improve the responses and fill context gaps even before we had comprehensive evals. In production, the API and UI return the trace identifier with the response, so feedback can lead directly to the trace for in-depth analysis.
Part 4: Keeping quality in the release path
Evaluation is not only for experiments with LLM orchestration, models, prompts, or context changes. Changes to log selection, trimming logic, or a source system can change what the model sees and therefore change extraction quality. We use the same evaluation harness and evaluators throughout local development and Pensive AI deployment so those effects are visible before users encounter them.
For each pull request, the code branch is deployed to an isolated preview environment and a CI job runs an experiment with our evals against that isolated environment. Precision and inclusion thresholds gate the merge. The job also runs after a build is promoted, catching interactions that were not present when individual pull requests were evaluated in isolation. If the promoted build misses its thresholds, it alerts the on-call engineer to investigate or roll back.
The evaluator runs on a schedule against production to surface gradual drift or external changes. Developers can also trigger evaluation experiments remotely while iterating, using the same deployed-environment path as CI. Consistent infrastructure makes the results comparable across local iteration, preview builds, promoted builds, and production.
Part 5: Looking for feedback signal from users
The system is only valuable if it reduces the amount of manual debugging and log chasing done by our users. We wanted a measurement that was available from product telemetry and close to the behavior Pensive AI was designed to change. Manual debug-log exploration became one of the main signals. The Pensive interface already offers links to logs for a failed step attempt. To estimate whether the feature reduced raw-log hunting, we compared failed step attempts where someone interacted with Pensive AI with those where no one did. The second group averaged 321% more manual debug-log exploration actions. This observational comparison does not establish causality or a specific reduction in mean time to resolution, but it suggests that focused diagnosis often allows practitioners to inspect fewer raw logs. Programmatic MCP access also lets teams build operational flows with their terminals or custom agentic experiences around the same diagnosis service.
Another more explicit signal we collect is thumbs up and thumbs down on the debugging response. These feedback signals land directly alongside the internal traces so we can quickly associate feedback to the complete log of inputs/outputs and the path the AI took to arrive at the answer. More importantly, we created an internal business-hours alerting system where each thumbs down response sends us a Slack alert. This was valuable to resolve a lot of rough edges and context gaps initially in the product and is helping us improve the product continuously even today.

Part 6: Future Work building on these foundations
Pensive AI is the foundation for more work that goes beyond diagnosis and recommended remediations. We are exploring “Pensive AI Deepthink,” an opt-in managed agentic experience with higher cost and latency. It builds on the precise, low-latency analysis produced by Pensive AI Debugging and gathers additional context from internal documentation, manuals, past Slack discussions, source code, read-only warehouse queries, and Spark execution details. A routing step directs the investigation to the appropriate skill, such as Spark failures, data-audit failures, or late workflows. This keeps the shared starting point stable and deterministic while allowing the investigation to become more specialized, further improving recall. We are also in talks with the Spark team to leverage Pensive AI responses for failure debugging analysis and A/B comparison for upcoming platform migrations.
We also plan to invest in “Pensive AI Error Classification”. Hard-to-maintain legacy rules also classify failures by restartability and by whether they are platform or user caused. Maestro acts on those labels and changes its retry behavior accordingly like failing fast or tuning the default number of retries (platform errors are tried more than user errors by default). Error classifications must meet a higher bar than error extractions and debugging alone. Moving the majority of the error classifications to AI would let us retire most of the remaining rule-maintenance burden and avoid retrying unknown failures that were unlikely to succeed. For restartability, the costliest error is classifying a recoverable failure as non-restartable, because Maestro will turn a step that could have been recovered into a hard failure, leading to end user alerts and manual remediations. The evaluation framework will need to be tuned around that outcome before AI-enabled classifications can take on those decisions. Equally important, we must avoid retrying non-transient failures that will not succeed, because retries waste compute and delay data pipelines.
Our work on Pensive AI in Data Platform addresses concrete user pain points in debugging, improves maintainability for platform owners, and improves precision and recall for error extraction. Pensive AI extends coverage to failures that legacy Pensive previously classified as unknown. Reliability comes from the system around the AI model calls: bounded error extractions, compatible client data contracts, responsive delivery, reproducible evaluations, caching and tracing, and release controls that keep quality and cost visible as the platform changes.
Part 7: Acknowledgements
This work would not have been possible without our stunning colleagues across Data Platform and our Data Engineering and Analytics partners who provided early feedback. While we cannot name everyone individually, we are deeply grateful for their contributions, which shaped both response quality and the product experience. Special thanks to Adith Swaminathan for advising on evaluations and inference improvements and acting as a thought partner. Thank you to the Orchestration team members, especially Deepak Ramalingam and Malcolm Ramirez for their additional engineering contributions.
For dogfooding and integrating Pensive AI as part of user support, we also thank the engineering support organization (ESO Team). We also thank Abhishek Kapatkar (Data Developer Experiences Team) and Emmie Dong (Data Movement Team) for their feedback to improve the Pensive AI integrations with the rest of the ecosystem. Special thanks to the AI Platform team (Kyle Ahn, Rob Mann) and Braintrust Team for helping with distributed tracing, evals infrastructure and model gateway runtime support.
We also thank Kamran Masood, Ely Spears, Ronald Leung, Anjali Norwood, Prashanth Ramdas, Charles Smith, Harikrishna Menon and other leaders for their constructive feedback and support on the Pensive AI project.
Making Failure Diagnosis Reliable with Pensive AI was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 Netflix Tech(Making Failure Diagnosis Reliable with Pensive AI),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。