11-19 更新 实时
首页/ 技术栈/ 正文

面向移动信息流与营销的多比例视频缩放

技术栈 来源:Netflix Tech 2026-10-27 00:01 0 阅读 机器翻译
面向移动信息流与营销的多比例视频缩放

Serdar Cellat、Yuchen Xie、Bernadette Marcu、Blaid Burgess、Cody Jarrett、Carlos Pereira、Qing Shi、Samuel Carvajal、Manuel Pamplona、Dorothy Jung、Feiko Ritsema、Benjamin Klein、Alex Woo、Eugene Lok、Anna Pulido、Rebecka Swe、Sanford Holsapple、Ranjith Kumar Shetty

在 Netflix,我们使命的一个关键部分是确保我们的内容在任何设备上都呈现出惊艳效果。竖屏格式对 Netflix 越来越重要,因为我们在社交媒体账号上拥有 1.2B 关注者,并且最近推出了 Clips——Netflix 移动应用中的竖屏视频流。传统上,制作竖屏素材意味着剪辑师和制片人要在编辑工具中手动重新构图横屏视频,而随着我们的内容库和使用场景增长,这一流程无法规模化。为了大规模支持我们的应用内信息流和外部营销活动,我们构建了一套自动化多比例视频裁剪系统,它紧密集成到我们的内部制作工具中,能以最少的人工投入生成高质量的竖屏及其他宽高比版本。

将电影感 16:9 素材转换为适合移动端的宽高比,远不只是简单居中裁剪。电影制作通常涉及动态的角色走位和复杂的摄影机运动。我们需要一个智能的“虚拟摄影机”,它能够跟随关键动作,对每个场景的构图方式做出响应,并生成像原始镜头一样具有刻意构图意图的裁剪效果。

我们将视频尺寸调整视为一个跟踪与时序优化问题,以此应对这一挑战。我们的流水线分三个主要阶段处理每个素材,旨在一次处理中生成 9:16、4:5 和 1:1 的裁剪版本,而不会产生额外的计算开销。

首先,我们使用基于神经网络的镜头边界检测器将视频切分为原子镜头。在很多情况下,一个镜头中会有多个人物在画面两侧对话。为更好地处理这类场景,我们采用基于主动说话人检测(Active Speaker Detection)的方法,将单个连续镜头在逻辑上拆分为多个子镜头。这样一来,随着对话推进,我们的虚拟摄像机就能进行切换并重新聚焦到不同的说话人身上。我们的主动说话人检测借助内部的人脸检测器和 LipFlap,可靠地识别真人实拍与动画作品中的活跃说话人。

在定义好镜头和子镜头之后,我们会识别每一帧中最重要的区域。我们将人脸检测结果与视觉显著性图相结合,筛选并确定应当优先对焦的人脸。在有多个人物的场景中,我们进一步利用活跃说话人状态、相对大小等信号对人脸进行排序。我们的模型将这些线索与镜头级别的内容理解聚合起来,为虚拟摄像机建立连续的“对焦区域”。

在确定对焦区域之后,接下来的挑战是决定裁剪窗口如何随时间移动。严格逐帧跟踪对焦区域会导致画面抖动、分散观众注意力,因此我们应用了一套启发式规则,考虑对焦区域在一个镜头过程中的移动情况。当其位移较小时,我们锁定一个静态裁剪窗口,持续覆盖相关区域。当出现明显运动时,我们计算动态跟踪窗口并对其路径进行平滑处理,使任何平移都显得有意为之、富有电影感。

我们将该缩放算法直接集成到内部的 Clips 工具中,因此剪辑师和制片人可以在现有工作流中直接使用它。借助该界面,他们能自动生成同一段素材的多个宽高比版本,并实现智能重新构图。一个统一的应用内模块提供了直观的镜头重新构图控件,让用户可以查看算法的裁剪结果,并在需要时快速进行针对性调整。

对于更复杂的镜头或定制化营销活动,制片人可以将缩放参数导出为 XML,并直接载入 Adobe Premiere 等外部时间线编辑软件。这样一来,代理机构和外部合作伙伴可以直接以算法的构图决策为起点,而不必手动重新裁剪,也无需依赖 Premiere 的 Auto Reframe 之类的工具。在我们的内部测试中,该缩放算法在一组精选评测集上表现优于 Auto Reframe,因此 XML 导出实际上是把质量更高的裁剪结果带入了外部工作流。通过将自动缩放与人工精修集中到同一处,这条流水线在保留创作控制权的同时,显著缩短了制作时间、减少了人工投入。

我们在多比例视频缩放上的工作,是让内容创作更具可扩展性、同时不牺牲创作意图这一更大目标中的一步。通过将算法驱动的重新构图与便于剪辑师使用的工具相结合,我们减少了团队花在重复性缩放任务上的时间,也让他们能更快地在移动端信息流和营销活动中展开尝试。随着我们继续将这些能力拓展到新的格式和工作流,我们的目标始终不变:让创意人员有更多时间去讲好故事,并为我们的会员和粉丝带来始终如一的高质量观看体验。

《面向移动信息流与营销的多比例视频缩放》最初发表在 Medium 上的 Netflix TechBlog,在那里人们正通过高亮并回应这篇文章来继续这场对话。

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Multi‑Ratio Video Resizing for Mobile Feed and Marketing)

Serdar Cellat, Yuchen Xie, Bernadette Marcu, Blaid Burgess, Cody Jarrett, Carlos Pereira, Qing Shi, Samuel Carvajal, Manuel Pamplona, Dorothy Jung, Feiko Ritsema, Benjamin Klein, Alex Woo, Eugene Lok, Anna Pulido, Rebecka Swe, Sanford Holsapple, Ranjith Kumar Shetty

Introduction

Clips in the Netflix mobile app.

At Netflix, a crucial part of our mission is ensuring our content looks stunning on any device. Vertical formats are increasingly important for Netflix, as we have 1.2B followers across our social media accounts, and have recently rolled out Clips, our vertical video feed in the Netflix mobile app. Traditionally, creating vertical assets meant editors and producers manually reframed horizontal videos in editing tools, a process that doesn’t scale as our catalog and use cases grow. To support both our in‑app feed and external marketing campaigns at scale, we built an automated multi‑ratio video cropping system, tightly integrated into our internal production tools, that can generate high‑quality vertical and other aspect‑ratio variants with minimal manual effort.

Our Video Resizing Approach

Converting cinematic 16:9 footage into mobile‑friendly aspect ratios takes more than a simple center crop. Filmmaking often involves dynamic character blocking and complex camera movement. We need an intelligent “virtual camera” that can follow the key action, respond to how each scene is composed, and produce crops that feel as intentional as the original shot.

We address this challenge by framing video resizing as a tracking and temporal optimization problem. Our pipeline processes each asset in three main stages, designed to produce 9:16, 4:5, and 1:1 crops in a single pass, without incurring additional computational overhead.

An overview of the three-stage process converting horizontal footage into mobile aspect ratios.

Shot Detection and Splitting

First, we segment the video into atomic shots using a neural‑network–based shot boundary detector. In many cases, shots contain multiple characters conversing on opposite sides of the frame. To better handle these scenarios, we use an Active Speaker Detection based approach to logically split a single continuous shot into multiple sub‑shots. This allows our virtual camera to cut and re-focus on different speakers as the conversation flows. Our Active Speaker Detection leverages an internal face detector and LipFlap to reliably identify active speakers in both live‑action and animated titles.

Shot splitting via Active Speaker Detection. The magenta box represents the dynamic vertical crop.

Focus Region Discovery

Once shots and sub‑shots are defined, we identify the most important region in each frame. We combine face detections with visual saliency maps to filter and prioritize the faces that should be in focus. In scenes with multiple characters, we further rank faces using signals such as active speaker status and relative size. Our model aggregates these cues with shot‑level content understanding to establish a continuous “focus region” for the virtual camera.

Blue box shows the focus region around the primary character for reframing.

Temporal Reframing

Once we’ve established the focus region, the next challenge is deciding how the crop should move over time. Tracking the focus region on a strict frame-by-frame basis would produce a shaky, distracting viewing experience, so we apply a set of heuristics that consider how the focus region moves over the course of a shot. When its displacement is small, we lock a static crop window that consistently covers the relevant area. When there is significant motion, we compute a dynamic tracking window and smooth its path so that any panning feels intentional and cinematic.

The vertical crop (magenta box) follows the character’s movement with a smooth panning path.

Tooling and Workflow Integration

We integrated the resizing algorithm directly into our internal Clips tool, so editors and producers can access it as part of their existing workflow. The interface enables them to automatically generate multiple aspect‑ratio versions of a clip with smart reframing. A unified in‑app module provides intuitive shot‑reframing controls, allowing users to review the algorithm’s crops, and quickly make targeted adjustments where needed.

For more complex shots or custom campaigns, producers can export the resizing parameters as XML and load them directly into external timeline editors such as Adobe Premiere. This allows agencies and external partners to start from the algorithm’s framing decisions instead of recreating crops manually or relying on tools like Premiere’s Auto Reframe. In our internal test, our resizing algorithm outperforms Auto Reframe on a curated evaluation set, so the XML export effectively brings those higher‑quality crops into external workflows. By bringing automated resizing and manual refinement into a single place, this pipeline significantly reduces production time and manual effort while preserving creative control.

Video resizing interface with reframing controls

Conclusion

Our work on multi-ratio video resizing is one step in a broader effort to make content creation more scalable, without compromising creative intent. By pairing algorithm-driven reframing with editor-friendly tools, we’ve reduced the time teams spend on repetitive resizing tasks and unlocked faster experimentation across mobile feeds and marketing campaigns. As we continue to broaden these capabilities to new formats and workflows, our goal remains the same: to give creatives more time to tell great stories, and to deliver a consistently high-quality viewing experience for our members and fans.


Multi‑Ratio Video Resizing for Mobile Feed and Marketing was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 Netflix Tech(Multi‑Ratio Video Resizing for Mobile Feed and Marketing),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗