AI Discover 的工程实践

作者:Varun Khaitan、Peng Ren、Gary Tang、Itay Teller、Atif Faheem、Nipun Kumar、Donnie DeBoer、Eduardo Lundgren、Matteo Rinaldi、Raghav Bakshi
我们最近推出了 AI Discover,这是我们 TV 应用中新增的一个标签页,一小部分会员可以在这里尝试用新方式找到想看的内容。多年来,我们的推荐一直在后台运转,根据会员的观看历史推断他们可能喜欢什么。AI Discover 从这种无声的排序转向可操控的发现。会员选择一个主题,比如“快速观看”,就会得到一行匹配的片名、一段说明它们为何契合的简短提示,以及几个可以接着尝试的更细分主题。从这里出发,他们可以继续收窄范围、前往新的方向,或调整所看到的内容。
构建 AI Discover 引出了许多团队都会认同的一个问题:如何在电视上借助生成式 AI 提供更好的会员体验,同时不牺牲速度、可靠性或安全性?我们的答案是新增一个智能层,也就是 AI Discover 背后的智能体框架(agent harness)。对于每个请求,它决定如何把 LLM 与我们的推荐系统结合起来。它建立在多年理解会员喜好的工作之上,也建立在我们近期将 LLM 应用于推荐和生成式体验的工作之上。这一层自身会规划每个请求,在那些系统上运行它,提供结果,并在离线环境中对其进行评估。
本文介绍了我们针对设计应对的挑战,以及在此过程中做出的关键决策。
Netflix 的每个页面都把推荐与规则结合在一起。我们挑选会员可能喜欢的片名,而可播放资格、分级和安全规则则决定哪些内容可以出现。基于 LLM 构建的体验需要同样的规则,还需要一些代码来决定何时调用 LLM 以及如何处理其输出。这段代码被称为智能体框架(agent harness),它是我们智能层的核心。
最简单的实现方式是把整个请求交给 LLM,让它来挑选影片、撰写标签并决定下一步做什么。然而,这会使我们在需要可管控结果的地方依赖概率性输出,因为 AI Discover 的每一轮交互最终都会呈现在某个人客厅的屏幕上。在我们这样的规模下,即使是罕见的失败,每天也会发生很多次。每一次缓慢的调用、无法观看的影片或不符合品牌调性的文案,会员都会看到。有七项挑战塑造了我们的设计:
当会员选择一个主题时,TV 应用会把请求发送给 Page Server,后者为所有 Netflix 应用构建页面,并将该请求异步交给智能层。从高层来看,这个新层包含两个组件:控制器(controller)和执行器(executor)。
控制器决定运行什么。它会收集每个请求所需的上下文,包括一天中的时间等实时信号。随后它确定一个计划——既可以是已有的计划,也可以是由其规划器(planner)草拟的新计划——对其进行校验,并只从 Memory Hub(我们按会员存储记忆的存储库)中检索该计划所需的会员详情。这些详情可能包括由 MemberLLM 生成的会员偏好摘要——这是我们自研的用于个性化推荐的语言模型——或会员分享的反馈。关于 Memory Hub 和会员偏好摘要,我们稍后会详细说明。
计划是一张由已批准的步骤构成的图,每个步骤运行一项能力。能力是一个命名的工作单元,可以有多个版本,并归属于某个族类,如检索、排序、生成或打包。它会声明自己的输入与输出、它可以看到的上下文,以及失败时会发生什么。
能力声明让计划变得可验证。在执行之前,控制器会校验图:每个步骤都经过批准,每个输入都有生产者,没有循环,并且计划最终恰好收敛到一页。之后,控制器会在结果被提供之前对其进行检查。
执行器负责运行计划。各个步骤通过声明的契约相互传递带类型的输出,因此每个步骤都能准确收到它所请求的内容。相互独立的步骤并行运行,每个步骤都有自己的截止时间和相应的回退行为。有些步骤是经过优化的 LLM 调用,每次只执行一个任务,例如撰写行标题或建议下一批主题,并且它们只在经过验证的计划范围内行动。运行前的检查与运行期间的契约共同构成了受治理的执行,其中每一项操作都保持在代码设定的限制之内。
这两部分都会写入请求记录,也就是我们在离线环境中用于评估和改进系统的追踪数据。
每个挑战都促成了一项决策。以下是我们所做的选择,以及它为我们带来的收益。
我们围绕 LLM 重新构想了这一体验。更省事的做法是把它们当作排序器插入,维持当下的延迟预算,并在必要时进行预计算。AI Discover 建立在围绕会员可操控的主题展开的来回交互之上,这让 LLM 承担只有它们才能完成的工作:生成主题、解释各行的内容、总结品味,并帮助挑选适合该会员和当下时机的标题。会员用遥控器选择主题,因此完全无需输入。在电视上,这些主题和解释拥有各自的原生 UI 元素。在屏幕背后,LLM 可以起草计划,而异步服务为多步骤的智能体式编排提供了所需的时间。
这开启了我们的传统技术栈无法单独支持的全新体验。
要让推荐结果有据可依,就必须建立在那些已经了解我们内容目录和会员的现有系统之上。大多数能力都封装了复杂且经过验证的 Netflix 系统,例如搜索、大规模排序和内容目录元数据访问。其中一些能力未来可能会运行自己的智能体。
重塑这些系统需要细致的工作,因为智能体式编排挑战了分布式系统许多长期存在的假设。一项能力现在必须在无人设计过的组合中工作,服务于一个在运行时才决定调用它的调用方。每项能力都需要明确的目的和语义接口,而塑造它意味着决定有多少要合并进单个步骤、有多少留给规划器。粒度过细,规划器出错的方式就更多;粒度过粗,它编排更丰富计划的空间就会变小。
作为回报,规划器获得了可以以新方式混合搭配的构建模块,连接各项服务所需的人工接线也大大减少。由于计划只能使用这些模块,即便是临时起草的计划也依然扎根于我们的内容目录和会员偏好。
为了控制 LLM 的延迟和错误,我们会在任何执行开始之前先规划每一个请求。让智能体使用工具为人熟知的方式是循环:推理、调用工具、读取结果、再次推理。这种方式很灵活,但你事先无法判断它会走多少步,也无法判断它是否走错了路。用智能体的术语来说,我们的做法是“先规划后执行”(plan-and-execute)智能体:LLM 可以提出路径,但由代码决定它是否运行。
计划来自两个地方:我们已经测试过的成熟计划,以及 LLM 临时起草的计划。由我们控制的路由器决定提供哪一个。无论哪种情况,计划在运行前都要通过相同的检查。如果规划器速度慢,或者其计划未能通过检查,我们就会回退到经过测试的计划。
这实现了自适应规划。规划器可以为请求找到新的处理方式,我们也能针对每项任务在成本、延迟和质量之间取得平衡,从精简的单次调用方案到更丰富的多步骤方案。
今天对某项任务而言最佳的模型或能力,几个月后可能就退居次席,或者变得大材小用,因此我们把系统构建成可以在它们之间切换。我们会把每项任务分配给适合它的模型或能力版本。目前这是一个按任务设置的配置,我们可以实时覆盖调整。一个请求中只有部分步骤会用到 LLM。
这只有在切换足够容易时才成立。我们使用的所有 LLM 都位于同一个通用接口之后。在切换之前,每个候选方案都会通过自动化检查和 LLM 评判器,在相同的任务上被打分,因此每次替换都是一次有度量依据的变更。
系统的许多行为都存在于服务端的提示词和配置中,因此我们无需编写新代码或发布 TV 应用版本,就能替换模型、工具和能力。
升级变得廉价,实验变得容易。迁移到更好的模型、排序器或检索系统只是一次方案或设置的变更,而非一次迁移,因此每个组件都能持续改进,而无需大额投入。
推理速度不断提升,但会员的期望同样水涨船高:更丰富的回答、更多的上下文,以及页面上更多由 LLM 塑造的部分。什么算即时、什么值得等待,这条界线会不断移动,因此我们把异步路径构建为一种一等公民式的服务方式。
电视应用的大部分逻辑是同步的:按一下按钮,就返回一个页面。生成式任务打破了这一模式。若让一个请求在多个服务之间保持打开状态长达数秒,会占用过多线程。因此,涉及生成内容的请求是异步的。按键会立即返回一个确认,电视随即切换到“思考”状态,说明正在准备什么。与此同时,执行器在后台运行该计划。结果就绪后,服务器通知电视,电视再去获取结果。
生成速度慢不再拖慢其他任何环节。无论一轮交互耗时多久,服务链路都保持健康,而每一次按键仍能在屏幕上获得即时反馈。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Engineering AI Discover)
By Varun Khaitan, Peng Ren, Gary Tang, Itay Teller, Atif Faheem, Nipun Kumar, Donnie DeBoer, Eduardo Lundgren, Matteo Rinaldi, Raghav Bakshi
We recently introduced AI Discover, a new tab in our TV app where a small group of members can try new ways to find something to watch. For years, our recommendations have worked in the background, inferring what a member might enjoy from their viewing history. AI Discover moves from that silent ranking toward steerable discovery. A member picks a theme, say “Quick Watches,” and gets a row of matching titles, a short note on why they fit and a few narrower themes to try next. From there, they can keep narrowing, head somewhere new or tune what they see.
Building AI Discover raised a question many teams will recognize: how do you deliver a better member experience with generative AI on TV without sacrificing speed, reliability or safety? Our answer is a new intelligence layer, the agent harness behind AI Discover. For each request, it decides how to combine LLMs with our recommendation systems. It builds on years of work understanding what members enjoy, and on our recent work applying LLMs to recommendations and generative experiences. The layer itself plans each request, runs it across those systems, serves the result and evaluates it offline.
This post covers the challenges we designed for and the key decisions we made along the way.

Why an intelligence layer?
Every Netflix page combines recommendations with rules. We pick titles a member may enjoy, and eligibility, maturity and safety rules decide what can appear. An experience built on LLMs needs the same rules, plus code that decides when to call an LLM and how to handle its output. That code is called an agent harness, and it’s the core of our intelligence layer.
The simplest implementation hands the entire request to an LLM and lets it pick the titles, write the labels and decide what happens next. However, that leaves us relying on probabilistic output where we need governed results, because every turn in AI Discover ends on a screen in someone’s living room. At our scale, even rare failures happen many times a day. A member sees every slow call, unavailable title or off-brand line. Seven challenges shaped our design:
- Fitting LLMs into a product and architecture designed without them. Traditional recommenders rank titles based on past behavior. LLMs can do more: understand what a member is asking for and plan how to respond. Our existing experience would leave most of that unused.
- Keeping recommendations grounded. Every pick must be a real, available title from our changing catalog. Retrieval and rankers work in tandem with LLMs to choose titles that fit the member’s taste and intent.
- Working with slow, unpredictable LLMs. LLM calls are the slowest and most expensive part of the system. Tail latencies can spike, and the output may not be what you asked for. So we use LLMs selectively, treat what they return as untrusted input and test performance and quality before anything ships.
- Adapting to fast-changing models, tools and prompts. Modeling and engineering solutions keep evolving, so the product can’t depend on any single model, tool or prompt.
- Building for TV. Typing on a TV is hard, so members rely on the remote, and the app should respond to every press right away on widely varying hardware. TV apps also update slowly, so changes have to ship without an app release. A typical TV request returns in moments, while multi-step generation can take several seconds.
- Holding a safety and reliability bar at scale. Members always get a page, even when an LLM call fails or yields poor results. Their choices and our content standards hold on every request, and when something breaks, structured records show why.
- Learning from what members say alongside what we already know. Recommendations have always learned from viewing behavior. AI Discover also lets members tell us what they want, and that has to shape what they see next.
System Architecture
When a member picks a theme, the TV app sends the request to the Page Server, which builds pages for all Netflix apps and hands the request asynchronously to the intelligence layer. At a high level, this new layer has two components: a controller and an executor.
The controller decides what to run. It gathers the context needed for each request, including real-time signals such as the time of day. It then settles on a plan, either an established one or a new one drafted by its planner, validates it and retrieves only the member details that this plan requires from Memory Hub, our per-member memory store. These details may include member taste summaries generated by MemberLLM, our in-house language model for personalized recommendations, or feedback shared by the member. We talk more about Memory Hub and member taste summaries later.
A plan is a graph of approved steps, each running a capability. A capability is a named unit of work that can have multiple versions and belongs to a family such as retrieval, ranking, generation or packaging. It declares its inputs and outputs, the context it may see and what happens if it fails.
Capability declarations make a plan verifiable. Before execution, the controller validates the graph: every step is approved, every input has a producer, nothing loops and the plan ends in exactly one page. Afterward, the controller checks the result before it’s served.
The executor runs the plan. Steps pass typed outputs to each other through the declared contracts, so each receives exactly what it asked for. Independent steps run in parallel, each with its own deadline and appropriate fallback behaviors. Some steps are optimized LLM calls with one task each, such as writing row titles or suggesting next themes, and they act only within the validated plan. Checks before the run and contracts during it give us governed execution, where every action stays within limits set in code.
Both parts write to the request records, the traces we use offline to evaluate and improve the system.

Key Decisions
Each challenge led to one decision. Here’s what we chose and what it bought us.
1. Design the product and architecture around LLMs
We reimagined the experience around LLMs. The easier path would have been to drop them in as rankers, keep today’s latency budgets and precompute where necessary. AI Discover is built on a back-and-forth around themes a member can steer, which gives LLMs work only they can do: generating themes, explaining rows, summarizing taste and helping choose the titles that fit the member and the moment. Members pick themes with the remote, so they never have to type. On the TV, those themes and explanations get their own native UI elements. Behind the screen, LLMs can draft plans, and asynchronous serving gives multi-step agentic orchestration the time it needs.
This opens up experiences our traditional stack couldn’t support on its own.
2. Make existing systems composable
Keeping recommendations grounded meant building on the systems that already know our catalog and members. Most capabilities wrap complex, proven Netflix systems such as search, large-scale ranking and catalog metadata access. Some of them may run their own agents in the future.
Reshaping them took careful work, because agentic orchestration challenges many long-standing assumptions of distributed systems. A capability now has to work in combinations no one designed, for a caller that chooses to call it at runtime. Each one needs a clear purpose and a semantic interface, and shaping it means deciding how much to combine into a single step and how much to leave to the planner. Too fine, and the planner has more ways to go wrong; too coarse, and it loses room to compose richer plans.
In return, the planner gets building blocks it can mix in new ways, and connecting services takes far less manual wiring. Because plans can only use these blocks, even one drafted on the fly stays grounded in our catalog and member taste.
3. Plan first, then execute
To contain LLM latency and errors, we plan every request before anything runs. The familiar way to let an agent use tools is a loop: reason, call a tool, read the result, reason again. It’s flexible, but you can’t tell up front how many steps it will take or whether it has gone down a wrong path. In agent terms, ours is a plan-and-execute agent: an LLM can propose the path, but code decides whether it runs.
Plans come from two places: established plans we’ve already tested, and plans an LLM drafts on the fly. A router we control decides which one is served. Either way, the plan passes the same checks before it runs. If the planner is slow or its plan fails a check, we fall back to a tested plan.
This allows for adaptive planning. The planner can find new ways to serve a request, and we can balance cost, latency and quality for each task, from a lean single-call plan to a richer multi-step one.

4. Stay model and capability agnostic
The best model or capability for a task today may be second-best, or overkill, in a few months, so we built the system to switch between them. We assign each task to the model or capability version that fits it. Today that’s a per-task setting we can override live. Only some steps in a request use an LLM.
That only works if switching is easy. All the LLMs we use sit behind one common interface. Before we switch, each candidate is scored on the same tasks by automated checks and LLM judges, so every swap is a measured change.
Much of the system’s behavior lives in server-side prompts and configuration, so we can swap models, tools and capabilities without new code or a TV app release.
Upgrades get cheap and experiments get easy. Moving to a better model, ranker or retrieval system is a plan or setting change, not a migration, so each component can keep improving without a large investment.
5. Make page generation asynchronous
Inference keeps getting faster, but member expectations rise just as quickly: richer responses, more context and more of the page shaped by LLMs. The line between what feels instant and what’s worth waiting for will keep shifting, so we built the asynchronous path as a first-class way of serving.
Most of a TV app is synchronous: press a button, get a page. Generative work breaks that pattern. Holding one request open for many seconds across services would tie up too many threads. So requests that involve generation are asynchronous. The press returns right away with an acknowledgement, and the TV switches to a thinking state that explains what’s being prepared. Meanwhile, the executor runs the plan in the background. When the result is ready, the server notifies the TV, which then fetches it.
Slow generation no longer slows anything else. The serving path stays healthy however long a turn takes, and every press still gets immediate feedback on screen.

6. Enforce safety in code
Some rules must hold no matter what an LLM writes: content and maturity standards, what each step can access, and what a member has asked us not to show them. Putting those rules in a prompt makes them probabilistic, so we enforce them in code. Generated text is cleaned up before it reaches the screen. The planner sees a stripped-down request, and member details are limited to what the validated plan needs.
For example, the planner decides which inputs each step needs, such as the profile’s maturity rating for retrieval. Instead of writing parameter values directly, it uses placeholders. After the plan passes validation, the controller fills them with values from the request, matched by declared type. This also protects against prompt injection.
Each request also leaves a structured record: the plan, each step and its outcome, and which rules applied. When a page misses, we can tell whether the plan, the data, a rule or the model was at fault. The same records feed our offline evaluation and replay.
Together, these rules and records let an experiment ship at the same safety and privacy bar as the rest of the product.
7. Remember what members tell us
A taste summary is a short profile of what a member enjoys, such as favorite genres, tone and recurring themes. MemberLLM generates it ahead of time from the member’s viewing history, which keeps that LLM work off the request path. It helps us further personalize their page, and steps that write themes and explanations receive it as context, so their wording reflects the member. We’ll cover MemberLLM in more detail in an upcoming post.
In AI Discover, members can also tune their recommendations and tell us directly what they want. Those choices apply right away and carry across visits, and they give us a new signal alongside what we already know about their taste. This is a first step toward remembering more about each member, from their preferences to the habits that show up over time and the corners of the catalog they like to explore. We keep it all in Memory Hub, our persistent, low-latency memory store, next to their taste summaries, and we can add new kinds of memory as we need them. We apply these memories to later requests where they’re relevant, together with real-time context.
The member holds the remote
In this version, members interact by selecting themes and using tuning options such as “don’t recommend.” There’s no free-text input or open-ended chat. That keeps the LLM’s job narrow by design.
- The usual protections still apply. Profile settings and our safety and privacy policies hold here as everywhere else.
- Generated text was tested before launch. We ran safety evaluations on theme labels, row titles and explanations to make sure they meet our standards.
What’s next
It would have been faster to let an LLM do everything and stop there. The intelligence layer gave us an AI-native product, grounded building blocks, adaptive planning, quicker experimentation, asynchronous generation, governed execution and agent memory for tunable personalization. While AI Discover is its first experience, we designed it with more in mind.
We’ve organized our efforts to answer two questions:
- How much do members want to shape their recommendations? Whether they’d rather steer or lean back, and how much, which kinds of themes help, and how we should blend what members tell us with our existing personalization systems.
- What tradeoffs does this technology bring? How cost and latency hold up in production as we ramp traffic, how long members will wait for generated results, and whether errors came from the model, the harness or our existing systems. Multi-turn sessions also need different evaluation and observability than single requests.
Longer term, our goal is a system that improves itself, starting from the record each request leaves behind. Today we replay past requests against a new prompt, model or rule to see what changes, and we judge each change with automated checks, LLM judges and human evaluations. Eventually, we want the system to propose improvements, test them safely and keep only what proves better, within our privacy and data-use policies. As LLMs improve, more of the work may move into them, and the intelligence layer lets us move that line deliberately.
This is part of a bigger shift. Recommendation systems have long learned quietly from what members watch and answered with ranked rows. LLMs let them listen and respond, and members may increasingly expect that. We’ll keep exploring what that means for discovery on Netflix.
Acknowledgements
We would like to thank Adi Suresh, Akash Katipally, Allan Johns, Becks Wood, Bhargav Mangipudi, Carrie Santiago, Chris Martin, Claire Campbell, Corey Waldin, Dana Sittler, Daneo Zhang, David Leen, Devon Bryant, Dharamendra Kumar, Dharini Chandrasekaran, Ebony Turner, Federico Roa, Gianfranco Cuoco, Guru Tahasildar, Haley Clanton, Haresh Chudgar, Ivan Provalov, James Schultz, JingXu Yang, Josie Shaw, Kamelia Aryafar, Kate Cummings, Kathleen Garrity, Kelley Robinson, Kevin Du, Kevin O’Connor, Kyla Moore, Kyle Lind, Laura Matsuura, Leah Cusella, Lingyi Liu, Lingyong Wang, Mallory Walkup, Matthew Pereira, Michael Galassi, Minsu Jang, Moumita Bhattacharya, Paran Jain, Paul Lee, Qusai Al-Rabadi, Scarlet Chen, Sergi Perez, Sriram Rao, Stephen Shaw, Sura Elamurugu, Varad Pathak, Yesu Feng, Zainab Mahar Mir, and Zhongli Cheng for their contributions to this work.
Engineering AI Discover was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 Netflix Tech(Engineering AI Discover),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。