11-19 更新 实时
首页/ 技术栈/ 正文

本体驱动的可观测性:Netflix 规模端到端知识图谱

技术栈 来源:Netflix Tech 2026-10-17 00:01 0 阅读 机器翻译
本体驱动的可观测性:Netflix 规模端到端知识图谱

Prasanna Vijayanathan, Renzo Sanchez-Silva

当 Jake Paul 和 Mike Tyson 在 2024 年 11 月登上拳台时,我们将这场比赛分发给了约 6500 万路并发流。在平常的一天里,我们的基础设施会接收超过 20 亿次请求,并每秒处理超过 3800 万个实时日志事件。在这种规模下,整体图景既装不进单个工程团队的脑子里,也放不进单个仪表盘。随着更多直播内容、播客和实时流,我们尽可能快地理解和更新系统,已成为提供良好用户体验的必需。在这种规模下、并在我们拥有的时间限制内理解系统,往往变成一项艰巨任务:在黑暗丛林中蒙着眼睛大海捞针。

这篇文章讲的就是那场寻找针的过程。

设想某周我们会遇到的一类回退。主屏幕上会显示一列列电影列表,而它开始多花约 1.5 秒来渲染。警报触发。然后我们开始寻找是什么导致了这个问题,以及我们该如何修复它?

碰巧的是,应用上的回退可能与系统中任何地方有关——新版本部署或 A/B 测试推出导致的客户端应用回退、基础设施中的减载、由于下游 6 跳处的服务发布而导致 GraphQL 查询返回错误数据,此时这些都是嫌疑对象。因此,回答“是什么导致了这个问题?”需要来自整个系统的洞察,而这涉及一丝不苟地梳理层层数据。

“是什么导致了这个问题?”

指标(Metrics)、事件(Events)、日志(Logs)和链路追踪(Traces),即业界简称的 MELT,正是此类排查所需的基础原料,它们各自大致回答了问题的四分之一。我们在这四大支柱上采集可观测性数据,并统一到一个可观测性数据层中。这让我们能够实时观测系统:用户正在经历什么、应用运行得如何,以及后端基础设施与服务正在处理什么、又在响应什么。

接下来的排查在每个层面上可能大致是这样的:

那么,排查一个问题、找出根因并完成修复,让我们的用户不再遇到该问题,需要多长时间?

在我们所举的这个例子里,是四个小时。其中只有很少一部分花在计算什么困难的东西上。大部分时间花在重建那些关联关系上——可观测性系统早已知道这些关系,但我们从未以人或机器能够直接读取和发现的形式把它们记录下来。

在 2023 年前后,人们的本能反应可能是去要更多数据或更好的仪表盘,尽管两者在边缘地带有帮助,但都未能解决让这类排查变慢的根本原因。

现在是 2026 年,所以我们或许会说“嘿,智能体”,但如果不能把正确的上下文交给智能体,AI 系统就会产生幻觉,尤其当团队处于值班状态、需要“立刻”修复时,我们更不能让不准确的智能体式分析把我们引向错误的道路。我们需要的是正确的那根针,而不是随便哪根针。

我们的每个系统内部自洽、对外隔离。Atlas 是我们的遥测平台,它了解各类应用及其时间序列。Radar 是我们的告警系统,它了解阈值和触发历史。incident.io 了解严重级别和响应人。PagerDuty 了解值班轮换。Spinnaker 了解部署了什么、何时部署。而我们的 Slack 历史记录则以一种任何查询引擎都无法触及的形式,记着上个季度是哪位工程师解决过非常类似的问题,以及他们的发现。

难点在于,它们对彼此谈论的对象并不一致。在一个系统里是应用(application)的东西,在第二个系统里是服务(service),在第三个系统里是集群(cluster),名称还可能因后缀、大小写约定,或只有少数人记得的重命名而不同(SME,听着耳熟吗?)。要跨这些系统做关联,就得靠人把映射关系记在脑子里,而在这类跨域排查中,这个人就成了瓶颈。端到端可观测性的难点,或许更应被理解为一个数据建模问题。

我们缺少的,是对运营世界中的实体形成一份明确的、机器可读的共识。

这份共识有个名字:本体(ontology)。本体是对类型、类型所携带的属性以及它们之间可能存在的关��的形式化规范。它的单位是三元组(triple),即用主语、谓语、宾语表达的一条事实。看下面这几条示例三元组。

api-gateway rdf:type ops:Application . api-gateway ops:ownedBy "Team Bedrock" . api-gateway ops:deployedIn "us-east-1" . INC-5377 rdf:type ops:Incident . INC-5377 ops:affects api-gateway . INC-5377 ops:severity "SEV 2" . 在这个例子中,我们已经编码了六条事实,并且可以用一次查询回答过去需要横跨三个工具才能回答的问题。

“哪些事件影响了 us-east-1 中由 Team Bedrock 拥有的应用?”这一条查询,就替代了分别去查 incidents、Spinnaker 和 Radar 的做法。

我们把这一层称为 The Observability Ontology(可观测性本体)。它充当模式、规则集和共识约定,标准化了不同系统对应用、服务、事件、部署等实体的理解方式。

为此,我们把它构建在 W3C 技术栈之上:用 RDF 做数据模型,用 OWL 做本体,用 SHACL 做校验,用 SPARQL 做查询。这些标准规范完善、实现广泛,使模型可以在不同图存储之间保持可移植性。

本体自身的数据以知识图谱形式存储,持久化到 QuipuDB——我们自研的 RDF 图数据库,一个基于 redb(基于 Rust 的嵌入式键值数据库)的分布式四元组存储。这里的数据来自对原始遥测、日志和事件讨论的解析,将其转化为三元组(主语—谓语—宾语事实)。

每次采集还会生成完整的 Turtle 序列化结果用于归档,使其人类可读、便于 diff,这在调试中的用处比我们预期的更大。

我们的本体划分为十二个命名空间,每个命名空间负责一个运维领域:

在构建过程中,有两条关于这个本体的决策至关重要,因此值得分享。

第一个决定是我们拒绝使用通用的 Service 类型。“Service”在我们的工具链中意味着四种不同的东西,因此应用程序最终只对应一种类型——ops:Application,并且在实体创建之前,每一个都会对照我们约 49,000 条记录的软件目录进行校验。像用户名、团队名、从网络拓扑中抓取来的依赖名这类未经校验的字符串,永远不会成为应用程序。我们还会对依赖边在两个方向上的数量设置上限,因为一个热门服务可能承载数千条反向依赖,否则它就会在图谱中占据主导地位。

第二个决定关乎我们选择建模的对象。事故聊天记录承载着事故背后的推理过程。尝试过又被放弃的假设、奏效的缓解措施、谁知道原因,这些上下文都在事故的 Slack 频道里,而一旦频道安静下来,这些知识就蒸发了。于是,一个包含 200 条消息、50 个人、15 个服务的事故频道会变成一个结构,把事故中混乱的运维过程转化为有序三元组:

构建这个图谱意味着要使用本质上具有随机性的工具。LLM 对讨论进行摘要。模糊匹配解析服务名称。概率引擎提出因果链。这些工具没有哪一个能确定地两次产出相同的结果,但我们系统最终产出的是一个确定性的图谱,它对 SPARQL 查询的每次回答都完全一致。

本体(ontology)正是通过其流水线式方法创造这种确定性的机制。我们把这条流水线视为一部驱动熵下降的阶梯,这里的熵指的是仍以非结构化形式被困住的信息所占的比例。例如,在这条流水线的起点,LLM 可以说任何话,但它无法输出违反类型系统的三元组。我们把输出空间约束得足够紧,于是随机过程产出了确定性的契约:

# The model may produce any prose. Its output MUST fit these predicates: category IN ["incident", "question", "discussion", "decision"] sentiment IN ["positive", "neutral", "negative", "urgent"] # Result — a deterministic triple: thread:123 slack:category "incident" . thread:123 slack:isIncidentRelated "true"^^xsd:boolean .有五层结构带我们沿这部阶梯下行。这部阶梯的任何一个单独步骤都是非确定性的。但聚合起来,整条链就变成确定性的。这种反转正是全部诀窍所在。下面我们逐一描述:

我们使用所谓的增强器(enricher)来降低知识图谱中的熵。数十个增强器在图上运行。每一个都接收带噪声的输入并输出带类型的三元组,而且每一个都会读取更早的增强器写入的内容。增强器的契约被刻意设计得很窄:

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale)

Prasanna Vijayanathan, Renzo Sanchez-Silva

When Jake Paul and Mike Tyson stepped into the ring in November 2024, we delivered the fight to roughly 65 million concurrent streams. On an ordinary day our infrastructure takes in more than 2 billion requests and handles upwards of 38 million real-time logging events every second. At that scale, the whole picture doesn’t fit into a single engineering team’s head or onto a single dashboard. With more Live content, Podcasts and real-time streams, our need to understand and update our systems as quickly as possible becomes imperative to provide a good user experience. Understanding the system at this scale and within the time constraints we have, often becomes a daunting exercise in finding a needle in a haystack, in a dark jungle, blindfolded.

This post is about that search for the needle.

The Four hours before a fix

Consider a regression of the kind we see in a given week. The home screen where you see the list of lists of movies, starts taking about 1.5 seconds longer to render. An alert fires. And then we go looking for what caused this issue and how do we fix it?

As it so happens, a regression on the application could be related to anywhere in the system — a client app regression from a new version deployment or A/B test rollout, a load shedding in the infrastructure, a GraphQL query responding with bad data because of a service rollout 6 hops downstream, are all suspects at this point. So, answering “what caused the issue?” needs insights from across the system, and this involves combing through layers of data in a meticulous manner.

The MELT Layer

“What caused the issue?”

Metrics, events, logs and traces, or MELT, in the industry’s shorthand, are the right raw materials for such an investigation, and each of them answers roughly a quarter of the question. We collect observability data across these four pillars, unified into a single observability data layer. This lets us observe our systems in real time, across what our users are experiencing, how our applications are doing and what our backend infrastructure and services are processing and responding to.

MELT — Metrics, Events, Logs and Traces, and how they are captured across the system.

The investigation that follows may look like this in each layer:

What each layer of MELT tells us and what it leaves out

The timeline of an investigation

So how long does it take to investigate an issue, figure out the root cause and fix it, so our users stop running into this issue again?

A timeline of an investigation: ~4 hours

In our chosen example, four hours. Very little of it went into computing anything difficult. Most of it went into reconstructing relationships that our observability systems already knew but we had never written down anywhere a human or machine could simply read and discover.

The disagreement underneath

The instinct here could have been, around 2023, to reach for more data or better dashboards, and while both help at the margins, neither addresses what makes these investigations slow.

It’s 2026 now, so we might say, “hey, agents”, but without the right context delivered to the agents, AI systems hallucinate, and especially when teams are on-call and need a fix “now”, we cannot let inaccurate agentic analysis lead us down the wrong path. We need the right needle, not any needle.

Each of our systems is internally coherent and externally isolated. Atlas, our telemetry platform, knows applications and their time series. Radar, our alerting system, knows thresholds and firing history. incident.io knows severities and responders. PagerDuty knows rotations. Spinnaker knows what shipped and when. And our Slack history knows, in a form no query engine can reach, which engineer solved something very similar last quarter and what they found.

The difficulty is that they do not agree on what they are talking about. A thing that is an application in one system is a service in a second and a cluster in a third, with names differing by a suffix, a casing convention, or a rename only a couple of people remember (SMEs, ring a bell?). Joining across them requires a human carrying the mapping in their head, and that person becomes the bottleneck in such cross-domain investigations. The hard part of end-to-end observability is probably better understood as a data modelling problem.

What we lacked was an explicit, machine-readable agreement about the entities in our operational world.

The Ontology

This agreement has a name: an ontology. Ontology is a formal specification of types, the properties they carry, and the relationships they may have. Its unit is the triple, one fact expressed as subject, predicate, object. Consider the following example triples.

api-gateway   rdf:type       ops:Application .
api-gateway ops:ownedBy "Team Bedrock" .
api-gateway ops:deployedIn "us-east-1" .

INC-5377 rdf:type ops:Incident .
INC-5377 ops:affects api-gateway .
INC-5377 ops:severity "SEV 2" .

In the example here, we have already encoded six facts, and can answer in one query, a question that used to span three tools.

“Which incidents affected applications owned by Team Bedrock in us-east-1?” is a single query in place of looking up incidents, Spinnaker and Radar.

We call our ontology layer, The Observability Ontology. It acts as the schema, rule set and shared agreement that standardizes how different systems understand entities like applications, services, incidents, deployments, etc.

We built it on the W3C stack for this: RDF for the data model, OWL for the ontology, SHACL for validation, and SPARQL for querying. These standards are well specified and widely implemented, which keeps the model portable across graph stores.

The data from the ontology itself, is stored as a knowledge graph, persisted to QuipuDB, our homegrown RDF graph database, a distributed quad store based on redb, a Rust-based embedded key value db. Data here is generated by parsing raw telemetry, logs, and incident discussions into triples (subject-predicate-object facts).

Every harvest also produces a full Turtle serialization for archival, making it human-readable and diff-friendly, which has proven more useful for debugging than we expected.

Our ontology is organized into twelve namespaces, each owning one operational domain:

There are two decisions about this ontology that mattered while we built it, so they’re worth sharing.

The first is that we refused a generic Service type. “Service” means four different things across our tooling, so applications resolve to exactly one type, ops:Application, and every one of them is validated against our software catalog of roughly 49,000 entries before the entity is created. Unvalidated strings like usernames, team names, dependency names scraped from network topology, never become applications. We also cap dependency edges in both directions, because one popular service can carry thousands of reverse dependencies and would otherwise dominate the graph.

The second decision concerns what we chose to model. Incident chat carries the reasoning behind an incident. The hypotheses tried and discarded, the mitigation that worked, who knew why, is all context in the incident’s Slack channel and that knowledge evaporates when the channel goes quiet. So a 200-message, 50-person, 15-service incident channel becomes a structure, converting the operational chaos of an incident into ordered triples:

Creating determinism from non-determinism

Building this graph means using tools that are inherently stochastic. LLMs summarize discussions. Fuzzy matching resolves service names. A probabilistic engine proposes causal chains. None of these produce the same output twice with certainty, but what comes out at the end of our system is a deterministic graph that answers a SPARQL query the same way every time.

The ontology is the mechanism that creates this determinism through its pipelined approach. We think of the pipeline as a ladder that drives entropy down, where entropy means the proportion of information still trapped in unstructured form. For example, at the start of this pipeline, an LLM could say anything, but it cannot emit a triple that violates the type system. We constrain the output space tightly enough and a stochastic process yields a deterministic contract:

# The model may produce any prose. Its output MUST fit these predicates:
category IN ["incident", "question", "discussion", "decision"]
sentiment IN ["positive", "neutral", "negative", "urgent"]

# Result — a deterministic triple:
thread:123 slack:category "incident" .
thread:123 slack:isIncidentRelated "true"^^xsd:boolean .

There are five layers that get us down this ladder. Any single step of this ladder is non-deterministic. But in aggregate, the chain becomes deterministic. That inversion is the whole trick. We describe them below:

The five layers to create determinism from non-determinism

Enrichers: Agents as entropy reducers

We use what we call, enrichers, to reduce entropy across the knowledge graph. Dozens of enrichers run over the graph. Each one of them takes noisy input and emits typed triples, and each reads what earlier enrichers wrote. The contract for an Enricher is deliberately narrow:

class Enricher:
async def _enrich_async(self, ...):
raw = await self.fetch(...) # 1. read noisy input
result = self.analyze(raw) # 2. process (may be stochastic)
graph.add((uri, RDF.type, OPS.X)) # 3. emit TYPED triples
graph.add((uri, OPS.pred, Literal(result.value)))
return EnrichmentResult(success=True, triples_added=count)

Three invariants that are standard across the enrichers:

  • every entity carries an rdf:type,
  • every URI follows a single naming scheme, and
  • the catalog gates every ops:Application.

Two operational choices are worth naming here, as well. The enrichment pipeline runs sequentially — load the graph, add triples, and save.

This sequencing removes an entire category of race conditions at a cost in wall-clock time we are happy to pay. And calls out to external systems like metrics, logs, events and the rest could be best-effort, so a slow downstream API doesn’t block the graph harvest, albeit some degradation. To that end, we are also working to translate these separate systems into the ontology, such as the Metrics Ontology and the Alerts Ontology, so everything could be queried from the Ontology itself.

The Correlation Engine: finding the correlation chains

The correlation engine assimilates the vast number of signals from diverse input sources and condenses them into causal or correlatory chains of events. Today, the engine reads thirteen signal types, including log errors, metric anomalies, slow and failed traces, alerts, incidents, deployments, config changes, experiments, autoscaling, commits, end-to-end bottlenecks, and error patterns. It then searches for paths among them. Some ordered pairs are known to be causal and carry a high prior weight. For example, deployment before log error, config change before metric anomaly, failed trace before incident, and so on. Others, which show high correlation are considered correlation chains. Such chains are scored on temporal proximity and severity, inside a ten-minute window, with anything under 0.6 confidence getting discarded.

The Correlation Engine

The role of the engine here is to propose with evidence, not to make conclusions. Hence, the engine stores its chains along with the graph records that were used to create them.

A 0.87 chain in the above example, is a strong hypothesis for a human or an agent to check, and the graph records the confidence alongside the claim so that consumers can understand why a chain is what it is and weigh their options to make decisions.

The knowledge flywheel

Each harvest cycle turns the same four steps, and each rotation leaves the graph richer for the next. This is the Knowledge Flywheel, leading to continuous enrichment and improvement of the context about our system, stored in the knowledge graph.

The Knowledge Flywheel

Observe collects the signals. Enrich adds typed triples. Infer finds patterns invisible in the raw data. Adapt feeds knowledge back, so that established patterns become detection rules. The compounding result of the four phases leads to a self-improving system. A causal chain discovered in one incident becomes a known pattern for the next, and the ontology gradually stops being a schema and starts being a memory.

Consider the example above. When we first see an incident in api-gateway caused due to a timeout and resolve it, we convert this knowledge into entries to the ontology. This becomes encoded knowledge. As we go through multiple cycles of this encoding, the graph starts learning patterns. After N cycles, patterns become wisdom — “api-gateway deploys cause 40% of SEV-2s” and “30% of deployment issues in api-gateway come from Connection Timeouts”. Surface this as an insight and we proactively start reducing incidents, not just react quickly to new ones.

Putting the graph to work

On top of the graph sits AutoRCA, our internal agentic AI solution, built with a set of specialist agents coordinated around one question. Walking the home page regression from the top of this post, the agentic system’s reasoning chain might look something like this:

Each specialization to the inputs is built by people who understand their respective domain deeply. That division of labour only works if the specialists can pass findings to one another without having to agree on the terms of communication or even have knowledge of each other. The ontology acts as the interface. Each specialist learns one vocabulary — the graph’s. A team writing specialized use cases for how they use traces record their findings as triples about an ops:Application, the logs specialization reads triples about that same ops:Application, and neither needs to know the other exists. A new specialist learns the same vocabulary once and can immediately read everything the other six have written.

Set against the investigation we opened with, the change is narrower than it might sound. The alert still fires at 3:30pm, an engineer still has to weigh the evidence, and the fix still has to be written by someone who understands the code. The hour or two that used to go into assembling context — finding the owning team, reconstructing what shipped, correlating information from different parts of the system, recalling that something similar happened last quarter — all that work is the part the ontology and the graph absorbs. We are early enough that the ranked hypotheses are sometimes wrong, which is why each one carries the evidence that produced it. But when the engineer now starts the investigation, they are already holding the picture that used to take half an afternoon to assemble.

If you want to build one of these

A fair amount of what we have built, we learned by getting it wrong first. Based on this experience, these are the principles we would carry into a second attempt.

  • Model identity before anything else. Choose one authoritative catalog per entity kind and validate against it before creating an entity.
  • Treat the ontology as code. Version it, review it, test it, and let changes arrive as pull requests. Treating the ontology as code lends itself to agentic development and lets teams scale and contribute to it.
  • Separate concepts from instances. Concepts change on a human timescale and instances change continuously, conflating them makes both harder to evolve.
  • Build on open standards. RDF, OWL, SHACL and SPARQL are specified, implemented many times over, and portable. A bespoke graph model ties a future consumer to one implementation.
  • Constrain stochastic tools with a type system. This is the principle we would defend hardest. Models are useful precisely where determinism is impossible, and a sufficiently narrow output contract converts their output into something a query engine can trust.
  • Make enrichment best-effort and ordered. A slow dependency should utmost degrade the graph, never block it. And ordering enrichers so each builds on the last buys more accuracy and that’s worth the tradeoff with parallelism that buys speed.
  • Model human reasoning alongside machine telemetry. The most valuable edges in our graph came from discussion. Telemetry records what the system did, discussion records what people understood and interpreted. The reasoning is more valuable to store and serve as wisdom when we need it.
  • Keep humans as the approvers. Let agents propose fixes, enrichers, and ontology changes, and route every proposal through a review process with humans in the loop.

What comes next

The graph makes three things possible. First, automatic root cause analysis, where a regression arrives with a ranked set of graph-backed hypotheses attached. Next, automated remediation, where a sufficiently confident diagnosis carries a proposed action like a targeted rollback, a traffic shift, a code change. And eventually infrastructure that heals itself, acting on a predicted failure before members feel it.

Alongside this, we plan to integrate the Observability Ontology with Upper, the domain-modeling language at the heart of Netflix’s Unified Data Architecture (UDA). Upper lets teams describe business and system domains as interconnected, machine-readable models and preserve their meaning and constraints as those models are projected into concrete schemas and systems. This integration would connect operational concepts such as applications, alerts, deployments, and incidents to the broader set of domains modeled across Netflix, giving engineers and agents a shared semantic layer for querying, validation, and reasoning. We will begin with a proof of concept that models incidents in Upper and use what we learn to determine where the two systems should converge.

Acknowledgements

None of this would exist without the teams who share their data, expertise, and patience with an early system. We want to specially acknowledge the efforts of our own team members who made this possible from very early on in this lifecycle, Kaela Marsheck and Aditya Gaur. We are also especially grateful to our collaborators in Netflix’s ontology working group, Peter Royal, Alex Hutter, and to Alexandre Bertails and members of the Unified Data Architecture (UDA)/Knowledge Foundations team for sharing Upper and their domain-modeling expertise, and for helping us explore how the Observability Ontology can become part of a broader semantic foundation at Netflix.


Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 Netflix Tech(Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗